How AREX Supports Recording and Replay Dubbo Custom Protocol

AREX can record requests of Dubbo2x and Dubbo3x protocols. This article will introduce a common solution that allows users to develop their own custom Dubbo protocols for recording and replay.

Background

AREX is an open source automated regression testing platform based on real requests and traffic. It achieve fast and effective regression testing through the traffic recording and replay with Java Agent technology and comparison technology.

Dubbo is a high-performance distributed services framework, which is based on RPC service invocation and service governance. It has features such as transparent remote invocation, load balancing, service registration and discovery, high scalability, and service governance.

Currently, AREX supports to record requests of Dubbo2x and Dubbo3x protocols. Test cases using the both native versions of Dubbo protocol is supported to replay in the Schedule Service, which is developed based on the Dubbo3x version. But user-customed Dubbo protocols, serialisation methods, and even more personalised modifications based on Dubbo (such as dubbox), are not supported.

For example, the Dubbo protocol used by one community user's company is based on the modification of Dubbox, and the package structure and serialisation and deserialisation methods of the protocol have been modified, which can almost be understood as a completely new protocol, and this specific type of Dubbo protocol cannot be recorded and replayed with AREX. In order to solve this problem, we forked a customised version of the main branch to record and replay this protocol, but this approach is only applicable to this particular protocol and lacks universality.

Considering that many community users may have similar problems in the future, we need a generic solution that allows users to develop their own custom Dubbo protocols to be adapted. In this article, we have documented the implementation ideas and details, and hope to provide reference for other users in the future.

Solutions

To understand better, let's first describe the workflow of the AREX service components.

When using the traffic recording feature of AREX, the AREX Java Agent records the data flow and request information of Java applications in the production environment. It sends this information to the AREX Data Storage Service, which imports and stores the data in a MongoDB database. When performing playback testing, the AREX Schedule Service retrieves the recorded data (requests) of the tested application from the database through the Data Storage Service based on user configurations and requirements. It then sends interface requests to the target verification service. At the same time, the Java Agent returns the recorded responses of external dependencies (external requests/DB) to the tested application. After the target service completes the request logic, it returns the response message. The Schedule Service compares the recorded response with the playback response to verify the correctness of the system logic. It then pushes the comparison results to the Analysis Service (Report Service), which generates a playback report for testing personnel to review. Throughout this process, the AREX Cache Service (Redis) is responsible for caching mock data and comparison results during the playback process, improving the efficiency of the comparisons.

The difficulty is that, since the schedule service implements replay through generalised calls, when the Dubbo version, protocol, and serialisation of a service does not match that of the service provider, then all sorts of problems can occur causing the call to fail.

AREX wants to minimise interference with the user service, so it does not want to care about the Dubbo configuration of the user service. In order to solve the problem of inconsistent between the Dubbo versions, protocols, and serialisation methods, you can consider handing over the sending of replay requests to the user service itself, and letting the user service do the adaptation itself.

Considering of this, we choose Java SPI (Service Provider Interface) to solve the problem. Java SPI is a standard service discovery mechanism in Java. It helps developers to extend and replace components in a programme without modifying the code.

Users only need to implement the replay interface defined by the schedule service, and then use the same framework for replay as for recording. During replay, the schedule service dynamically loads the interfaces implemented by the user through the Java SPI mechanism. This allows the user service to adapt to the Dubbo version, protocol, and serialisation method. For the native Dubbo protocol, AREX implements an extension and loads it in by default.

Code Analysis

SPI Definition

Invoker

public interface ReplayExtensionInvoker {
    /**
     * check caseType by InvokerConstants#XXXCaseType.
     */
    boolean isSupported(String caseType);

    ReplayInvokeResult invoke(ReplayInvocation invocation);

    default int order() {
        return 1;
    }
}

Here we refer to the Dubbo source code for Invoker and define an interface called ReplayExtensionInvoker, which defines two methods: isSupported and invoke, which are used for checking the supported caseType type and performing the playback operation, respectively. The argument to the invoke method, Invocation, is a packaged request containing various information about the request, such as method names, parameter values, and so on. The return value of the invoke method is an InvokeResult object that contains information about the result of the playback. There is also a default method order defined in the interface which specifies the order of execution of the implementation class.

• isSupported()

The caseType indicates the protocol type of the playback use case, which can be expressed using the predefined XXXCaseType constant in the InvokerConstant class. Users need to bind (match) their own implementation of a replay extension to the specified caseType by implementing the isSupported method of the ReplayExtensionInvoker interface. In this way, when AREX replays, it will automatically match the corresponding replay extension according to the protocol type in the test case, so as to implement the replay function of the corresponding protocol type.

• order()

The same caseType may implement multiple extensions, which are loaded here in descending order by specifying the order in which they should be loaded first.

Invocation

public interface ReplayInvocation {
    /**
     * eg: dubbo:127.0.0.1:20880
     * protocol + host + port
     */
    String getUrl();

    Map<String, Object> getAttributes();

    ReplayExtensionInvoker getInvoker();

    void put(String key, Object value);

    /**
     * Get specified class item from attributes.
     * key: refer to InvokerConstants.
     */
    <T> T get(String key, Class<T> clazz);

    /**
     * Get object item from attributes.
     * key: refer to InvokerConstants.
     */
    Object get(String key);
}

Invocation is defined as an interface and the user needs to implement some methods in this interface in order to get the request parameters during replay.

• getUrl()

the protocol, host and port number of the returned request, refer to other protocols such as gRPC, Triple, Rest, etc. Url is required. We hope to support future extensions for other protocols to include Url as a separate parameter, consisting of protocol name + IP address + port number. For example: dubbo:127.0.0.1:20880.

• getAttributes()

The schedule service packages the other parameters of the request in a Map object with a key-value pair of type String, where the key needs to be the same as the name agreed upon in InvokerConstants. When implementing the replay extension, the user can get this Map object by calling the getAttributes() method, and then retrieve the corresponding parameter values from InvokerConstant based on the key name, so that they can be processed during the replay process.

Result

public class ReplayInvokeResult {
    /**
     * invoke result.
     */
    private Object result;

    private Map<String, String> responseProperties;

    /**
     * if invoke failed.
     */
    privdate String errorMsg;

    /**
     * if invoke failed.
     */
    private Exception exception;
}

The ReplayInvokeResult class represents the result of a request invocation and contains the following fields:

• result: the result of the request invocation, stored in a variable of type Object.
• responseProperties: some implicit parameters to be passed by the provider, stored in a Map.
• variable of type Map.
• errorMsg: a string storing the error message if the request call fails.
• exception: the object that stores the exception information if the request call fails.

Extension loading

During the loading process of the extension, we first get a URLClassLoader object by fetching the ClassLoader of the current class, and then call its addURL() method by reflection to load the extension implementation class.

The loadJar() method accepts a jarPath parameter, which indicates the path of the jar package where the extension implementation class is located, and then it gets a URLClassLoader object and a reference to the addURL() method by determining the current JDK version and obtaining a ClassLoader. Then, determine whether the current ClassLoader is of type URLClassLoader, if so, call the addURL() method by reflection to load the jar package, and add the classes in the jar package to the ClassLoader, so that you can dynamically call the methods of the extended implementation class when the application is running. If loading fails, then output an error message.

public void loadJar(String jarPath) {
       try {
           int javaVersion = getJavaVersion();
           ClassLoader classLoader = this.getClassLoader();
           File jarFile = new File(jarPath);
           if (!jarFile.exists()) {
               LOGGER.error("JarFile doesn't exist! path:{}", jarPath);
           }
           Method addURL = Class.forName("java.net.URLClassLoader").getDeclaredMethod("addURL", URL.class);
           addURL.setAccessible(true);
           if (classLoader instanceof URLClassLoader) {
               addURL.invoke(classLoader, jarFile.toURI().toURL());
           }    
       } catch (Exception e) {
           LOGGER.error("loadJar failed, jarPath:{}, message:{}", jarPath, e.getMessage());
       }
   }

How to use

Step 1: Develope SPI

1. Add pom Dependency

To develop an SPI extension, you first need to add the arex-schedule-extension dependency to your project's pom.xml file.

      <dependency>
           <groupId>com.arextest</groupId>
           <artifactId>arex-schedule-extension</artifactId>
       </dependency>

2. Implement SPI

Implementing the SPI extension requires an implementation of com.arextest.schedule.extension.invoker.ReplayExtensionInvoker.

Note that in the provider, the Arex-Record-Id in the Dubbo implicit parameter attachments is used to determine if the current traffic is replay traffic, and the resulting Arex-Record-Id is passed back via attachments as well.

Therefore, when implementing the ReplayExtensionInvoker interface, you need to get the Arex-Record-Id from the ReplayInvocation#attributes property and add it to attachments; and also add the Arex-Record-Id into the ReplayInvokeResult#responseProperties property so that these parameters are used correctly in the playback traffic.

An example of a default dubboInvoker is shown below:

package com.arextest.dubboInvoker;

import com.arextest.schedule.extension.invoker.InvokerConstants;
import com.arextest.schedule.extension.invoker.ReplayExtensionInvoker;
import com.arextest.schedule.extension.invoker.ReplayInvocation;
import com.arextest.schedule.extension.model.ReplayInvokeResult;
import org.apache.dubbo.config.ApplicationConfig;
import org.apache.dubbo.config.ReferenceConfig;
import org.apache.dubbo.rpc.RpcContext;
import org.apache.dubbo.rpc.service.GenericService;

import java.util.List;
import java.util.Map;

public class DefaultDubboInvoker implements ReplayExtensionInvoker {

    @Override
    public boolean isSupported(String caseType) {
        return InvokerConstants.DUBBO_CASE_TYPE.equalsIgnoreCase(caseType);
    }

    @Override
    public ReplayInvokeResult invoke(ReplayInvocation replayInvocation) {
        ReplayInvokeResult replayInvokeResult = new ReplayInvokeResult();
        try {

            RpcContext.getServiceContext().setAttachments(replayInvocation.get(InvokerConstants.HEADERS, Map.class));
            ReferenceConfig<GenericService> reference = new ReferenceConfig<>();
            reference.setApplication(new ApplicationConfig("defaultDubboInvoker"));
            reference.setUrl(replayInvocation.getUrl());
            reference.setInterface(replayInvocation.get(InvokerConstants.INTERFACE_NAME, String.class));
            reference.setGeneric(true);
            GenericService genericService = reference.get();
            if (genericService == null) {
                return replayInvokeResult;
            }

            Object result = genericService.$invoke(replayInvocation.get(InvokerConstants.METHOD_NAME, String.class),
                    (String[]) replayInvocation.get(InvokerConstants.PARAMETER_TYPES, List.class).toArray(new String[0]),
                    (replayInvocation.get(InvokerConstants.PARAMETERS, List.class)).toArray());

            replayInvokeResult.setResult(result);
            replayInvokeResult.setResponseProperties(RpcContext.getServerContext().getAttachments());
        } catch (Exception e) {
            replayInvokeResult.setException(e);
            replayInvokeResult.setErrorMsg(e.getMessage());
        }
        return replayInvokeResult;
    }

}

3. Create SPI Extension Files

Create a text file named com.arextest.schedule.extension.invoker.ReplayExtensionInvoker in the resources/META-INF/services directory of the project. The contents of the file are the fully qualified name of the extension implementation class, i.e. the Reference of the class that implements the ReplayExtensionInvoker interface.

4. Package jar package

Package the implemented SPI extensions into a jar package, including all relevant dependencies.

Step 2: Import AREX project

1. Jar package mapping to Docker image

First you need to map the jar packages to the Docker image, this can be done by modifying the docker-compose.yml file to map the local directory to the Docker image and add the jar packages to the corresponding directory on the local machine.

2. Modify the startup parameters of schedule service

Next, modify the startup parameters for the Schedule Service, again by modifying the docker-compose.yml file. Add the following startup parameters to the Schedule startup parameters.

-Dreplay.sender.extension.jarPath=/usr/local/tomcat/custom-dubbo/xxx.jar

xxx.jar is the name of the jar package packed in the previous step, and /usr/local/tomcat/custom-dubbo/ is the path to the jar package in the Docker image. The purpose of this startup parameter is to tell AREX where to find the jar package for the SPI extension.

Foresight

There are three points can be improved in the future:

未来还有三个方面可以进行改进：

Loading mode of jar packages

Currently, you need to manually configure the path of the jar package and modify the docker-compose.yml file, which is a bit cumbersome. In the future, we hope to be able to configure parameters and upload jar packages on the AREX platform.

Parameter convention

Currently, the management of parameters is implemented as hardcode in the code, and new accessors need to agree in advance and then update the SPI package. In the future, it is hoped that parameter configuration can be done on the AREX platform, so that new accessors can realise their needs through simple configuration.

Support for other protocols

Currently the invoker framework can directly support various custom extensions to the Dubbo protocol. If you need to access other protocols, such as gRPC, Triple, Rest, etc., you can also raise [issue] (https://github.com/arextest/arex/issues) for discussion in the community in order to extend the scope of the invoker framework.