About AREX

Real automated API testing with real data.

Introduction

Background

For newly launched simple services, testing can be accomplished through a combination of automated and manual testing. However, for complex online business systems that undergo frequent updates, it is crucial to ensure the accuracy of core business functions during modifications.

Traditional automation testing often requires significant human resources for test data preparation and script creation, and may not provide adequate coverage. To maintain the stability of online systems, both developers and testers face the following challenges:

• After development, it can be challenging to quickly verify locally and identify initial issues.
• Preparing test data, writing and maintaining automation scripts are time-consuming and may not provide adequate coverage.
• It is difficult to verify the data written to services, and testing may produce dirty data, such as in our core trading system, which may write data to databases, message queues, Redis, etc. This data is often difficult to verify, and the data generated by testing is also difficult to clean up.
• Online issues are difficult to reproduce locally, making it difficult to debug.

What is AREX?

AREX is a tool that automates regression testing by recording real traffic from the online environment to the test environment, solving the problem of regression testing. AREX uses Java's instrument to implement data collection and automated mocking without code intrusion. The intelligent mocking mechanism allows the test to run the code on the application being tested, without causing real external interactions (such as writing to the database or calling other services), and also perfectly supports testing of written interfaces (such as core transaction systems and inventory systems).

AREX supports various types of API testing, including:

• API testing, similar to Postman testing, with case setting, execution, and result assertions.
• Comparison testing, which sends the same request to different interfaces and compares the differences in the returned results, supporting both non-MOCK and AREX real data MOCK testing.
• Replay testing, which uses real production data for comparison testing.

How it works

We assume that the applications in production environment will normally respond to users' requests. By using the AOP approach, it saves the request parameters and return results, as well as some snapshot data of the execution process, such as the input and output of accessing the database and the input and output of accessing the remote server. Then the snapshot data is sent to the test machine (which with code changes) to complete a replay process. By comparing the snapshot data, the data of the backend requests and the returned results with the data of the real online requests, the differences are found and the problems of the system under test are identified.

• xxxTestCase: The collected data that used as a test case during Replay.
• xxxMock：Mock with collecte data during replay, instead of real data.
• xxxExpect and xxxReal：After the test, verify the recorded and replay data to discover hidden dangers in the code

Technical Principle

The Java Development Kit 1.5 introduced the java.lang.Instrument package, which provides the necessary tools for developers to dynamically modify classes within a system at runtime, thereby enhancing the functionality of the original class. Many contemporary tools, such as the open-source arthas and the monitoring tool SkyWalking, are built on this technology. AREX's data collection and automatic mocking are also rooted in this technology.

Advantages

Low Cost

No code intrusion, low integration cost. No need to write test cases, and massive online requests can also ensure high coverage. Staking code is simple enough for low performance loss.

Diversity Support

Supports validation of data written to the service, validation of database, message queue, Redis data, and even validation of runtime in-memory data. Does not produce dirty data during testing.

Stable running of test cases

Supports automatic data collection and mock for various mainstream technical frameworks, see: arex_java, and supports local time and cache, accurately restoring the data environment during production execution during replay.

Quickly reproduce online issues in local environment

Supports one-click local debugging, allowing for quick debugging of online issues in a local environment.

Safe and stable

Code isolation is also implemented in AREX, which also provides health management. During times of high system activity, AREX will intelligently reduce or turn off data collection frequency.

Good functional testing support

Support for test scripts and simple editing of the collected data to achieve fixed test cases, avoiding the need for extensive test data preparation.

Technical Implementation

We have implemented bytecode modification using the ByteBuddy library and have encountered various challenges in the process.

Trace Transfer

During data collection, AREX collects multiple pieces of data for the same request (e.g. request/response, requests and responses for other service calls), and we need to link these pieces of data together to form a complete test case. However, our application often uses an asynchronous framework and extensively uses threads, which makes it difficult to link the data.

Java Executors

Java and various frameworks provide many thread pool implementations, and we need to ensure that Trace data can be correctly transferred between threads. Therefore, we have modified Runnable/Callable as follows:

public class RunnableWrapper implements Runnable { 
  private final Runnable runnable; 
  private final TraceTransmitter traceTransmitter;
  private RunnableWrapper(Runnable runnable) { 
    this.runnable = runnable; 
    this.traceTransmitter = TraceTransmitter.create(); 
  }

  @Override public void run() { 
    try (TraceTransmitter tm = traceTransmitter.transmit()) { 
      runnable.run(); 
    } 
  }

  @Override public boolean equals(Object o) { 
    if (this == o) return true; 
    if (o == null || getClass() != o.getClass()) 
      return false;
    RunnableWrapper that = (RunnableWrapper) o; 
    return runnable.equals(that.runnable); 
  }

  @Override public int hashCode() { 
    return runnable.hashCode(); 
  }

  @Override public String toString() { 
    return this.getClass().getName() + " - " + runnable.toString(); 
  }

  public static Runnable get(Runnable runnable) { 
    if (null == runnable || TraceContextManager.get() == null) { 
      return runnable; 
    }

    if (runnable instanceof RunnableWrapper) { 
      return runnable; 
    } 
    return new RunnableWrapper(runnable); 
  }
}

Then, the code modifies various thread pools by replacing Runnable/Callable with Wrapper, and Wrapper ensures the correct transfer of Trace through TraceTransmitter internally.

ForkJoinPool

CompletableFuture and the default parallel stream processing of data sets use ForkJoinPool, which is also commonly used in heavily parallelized applications. This has significant differences from the typical thread pool implementation and requires separate handling. We have modified the ForkJoinPool task unit ForkJoinTask, which is more complex to implement than Runnable and is difficult to handle in a simple way. In order to not disrupt the original class structure (Agent on attach method does not support modification either), we did not add a field on this class to implement data transfer, but instead used a WeakCache for data buffering to ensure trace transfer between the task generation and execution threads.

Async

There are many asynchronous frameworks (Reactor, RxJava etc.) in the Java ecosystem, and many libraries also provide asynchronous implementations, such as lettuce provides both synchronous and asynchronous access to Redis. Different scenarios often require different solutions. For example, ApacheAsyncClient uses fixed running threads to listen for responses and initiate Callbacks. We need to ensure that Trace is correctly passed across threads in the entire process of calling, listening and callback. You can refer to TraceTransmitter for specific implementation.

Other

Version Management

Popular components often have multiple versions in use in different systems at the same time, and the implementation methods of different versions may vary greatly or even be incompatible. AREX also supports multiple versions (such as Jedis). We need to ensure that we can inject bytecode based on the correct version to avoid running errors. Bytecode injection is performed during class loading, so we must identify the component versions that the application depends on before these classes are loaded, and then match the versions during class loading to ensure correct code injection. You can refer to VersionMatch for part of the implementation.

Time Management

Many business systems are time-sensitive, and different time accesses often return different results. Therefore, we have also implemented the time Mock function. Because replay is executed in parallel, it is not appropriate to modify the machine time of the test machine (and many servers cannot modify the current time). Therefore, the time Mock is still implemented at the code level. During data collection, we record the current time for each case. During replay, we will mock System.currentTimeMills (many time underlying Java is through this method), and the time-sensitive methods related to the current time will get the correct time value through this method. This can ensure that the replay result is consistent with the collection.

Local Cache

Business applications may use various caches to improve runtime performance, and there may be significant differences in these data in different environments (syncing data between multiple environments is often a complex process). These data differences may result in completely different execution results. To avoid this problem, AREX also supports the collection and Mock of local cache data. With some simple configuration, data can be automatically collected and mocked. Of course, this function also supports Mock of various memory data.

Code Isolation and Interoperability

To ensure system stability, the AREX agent's framework code is loaded in a separate Class loader and is not interoperable with application code. To ensure that injected code can be accessed correctly at runtime, we have also modified the ClassLoader to ensure that runtime code is loaded by the correct ClassLoader (think SpringBoot's LaunchedURLClassLoader).

AREX Composition

AREX is composed of multiple modules, including Front, Schedule Service, Storage Service, Report Service, and data storage such as Mongodb, Redis.

When using the AREX traffic recording function, the AREX Java Agent will record the data traffic and request information of Java applications in the production environment, and send this information to the AREX data access service (Storage Service), which imports it into the MongoDB database for storage. When a playback test is required, the AREX scheduling service (Schedule Service) will extract the recorded data (requests) of the tested application from the database via the data access service according to the user’s configuration and requirements, and then send interface requests to the target validation service. At the same time, the Java Agent will return the response of the recorded external dependencies (external requests/DB) to the tested application, and the target service will return the response message after processing the request logic. Subsequently, the scheduling service will compare the recorded response message with the playback response message to verify the correctness of the system logic, and push the comparison results to the analysis service (Report Service) to generate a playback report for testing personnel to check. Throughout the process, AREX’s cache service Redis is responsible for caching mock data and comparison results during playback to improve comparison efficiency.

AREX UI

The AREX Front End is the front-end operator interface for AREX tools.

Schedule Service

The Schedule Service is responsible for sending case replay requests to the tested service and triggering result comparisons and dependency comparisons after the service responds.

Storage Service

The Storage Service is responsible for receiving and storing the actual data of requests, responses, and dependencies captured by Agent, and returning stored data as requested by Agent during replay.

Report Service

The Report Service is responsible for collecting and displaying test results and issues during the execution of replay testing.

Data Storage

The Redis storage service is responsible for caching data during replay;

The MongoDB storage service is responsible for storing recorded data and replay results.