API Testing Tool AREX

AREX combines Postman + Mock + Comparison Testing to achieve a closed-loop workflow from API debugging to API data Mock, to API automation testing and API regression testing. With AREX, developers and testers can work together to achieve more efficient software development and testing.

Background

Throughout the entire software development lifecycle, software testing is always present, including testing during the development process, integration testing before code merging, and continuous integration testing after deployment. In order to ensure comprehensive and stable software quality, both developers and testers need to continuously repeat the following testing tasks:

• API Testing

API testing is an effective way for developers to do quick self-test after completing development. Usually, you can use some interface debugging tools, such as Postman, ApiFox, etc., to send requests to the interface and observe the return results to debug and solve the problems in the interface implementation.

• API Automating Testing

Testers prepare test data, write test cases and use tools such as JMeter to write automated test scripts and automate the execution of test cases to ensure the quality and stability of the code.

• API Data Mock

Mock means imitation in English, and as the name implies, data mocking is to achieve testing purposes by constructing fake data. Using tools such as MockJS to mock API data can save time in data construction, which is crucial to improve the efficiency of regression testing in later stages.

• API Regression Testing

Regression testing is designed to ensure that the software can still function properly after modifications, new features, performance optimizations, etc.It is especially critical in rapid iterative development of software. In API regression testing, it is usually necessary to construct data, maintain data, test cases, and integrate with CI/CD processes.

For a development team, to meet the requirements of all stages of the development process mentioned above, at least four or five different tools are needed. The problems that arise in actual work can be imagined.

1. Increased complexity: Using multiple tools may lead to a more complex R&D process that requires more time and effort to coordinate and manage, and teams are unable to collaborate in the same system, resulting in a variety of untimely communications and inefficient collaboration.

2. High risk: Using multiple tools also means that data needs to be synchronized from one tool to another, which may increase the risk of errors.

3. High cost: Each tool has its own usage and features, which takes time to learn and master, and later it also takes more time to maintain and update the tools, especially in long-term projects.

4. Increased complexity of regression testing: For complex systems that require rapid iteration, frequent maintenance of data and use cases is required, either scripted or manual maintenance, which requires a lot of effort. When business complexity is high, the complexity of regression testing may increase, which may require more time and resources to complete.

5. "Automated" testing focuses on "automated" execution, but maintenance is still "manual" and the workload is very high.

Imagine if there was a tool that could achieve all the above functions, wouldn't that greatly simplify the R&D process, improve testing efficiency, and reduce maintenance costs and error rates? Today we are going to introduce such a tool: AREX.

What is AREX?

AREX is an open source automation test platform(https://github.com/arextest), combining Postman + Mock + comparison testing.

It not only provides API testing functions, but also records and stores request and answer data in the production environment through Java Agent bytecode injection technology, and replay the request and injected Mock data in the test environment, and stores new answers, so as to achieve automatic recording, automatic replay and automatic comparison, and provide convenience for API regression testing, realizing a closed-loop workflow from API testing to API data Mock, then to API automation testing and API regression testing. With AREX, developers and testers can each take what they need and work together to achieve more efficient software development and testing.

API Testing (Postman)

AREX not only supports most API testing features of Postman, but also provides powerful API test cases management features.

API Test Cases Management

AREX allows saving multiple API test cases for the same interface request under different scenarios. Each test case will automatically inherit the configuration of the current request, such as URL, request method, parameters, headers, body, and pre-request script. By inheriting the request configuration from the parent node, test cases do not need to redefine the request parameters and pre-request script of the API one by one. When test again in the future, you can directly run the test case, which reduces the workload and time of writing test cases.

AREX also provides test case management function. At present, test cases can be classified by Add Tag, and the function of searching test cases by tag will be added later to facilitate management. In addition, you can add descriptions to test cases to improve their readability and make it easier for collaborators to understand the purpose and expected results of test cases and to better grasp the test content.

Importing and Exporting Test Cases (Under Development)

Each testing team has its own testing habits, and it is very costly to migrate to a new testing tool. AREX will support the import and export of test cases in various formats, so that many test cases from the old tool can be migrated to AREX, without the need to create a new project.

Currently AREX is supporting the import of Postman exported test cases. If you want to support the import of test cases in other formats, you can submit Issue on Github to build with the community.

Saved test cases in AREX are also supported for export, currently Postman formatted cases (JSON format) are supported.

API Regression Testing (Mock + Comparison Testing)

AREX adopts Java instrument technology to realize data collection and automated mock without code intrusion, and the data collected during recording is used for mock during replay instead of real data access, which does not generate real external interaction (such as DB writing and other third-party service calls), avoiding the writing of dirty data during replay testing. It also supports automatic data collection and mock of various mainstream technology frameworks, including database redis, etc. It also supports Mock of local time, cache and other data, so that the data environment of production execution can be perfectly reproduced in the test environment during regression testing.

If users need further information about the Mock data, they can find the specified test case in the replay tab to view the Mock data and modify the Mock data.

API regression testing, no need to prepare test cases, test data, API ASSERT, just install AREX Agent, you can all get the latest requests from the production environment, the latest data, and use the comparison differences to verify the latest code:

• Support recording of production requests and responses
• Support recording of external requests and answers of the API, including http requests, data requests, redis requests, etc.
• Support automatic comparison of API response differences and request differences of external requests
• Support replay requests and MOCK external requests

Step 1: Install the AREX service

git clone https://github.com/arextest/deployments.git
cd deployments
docker-compose up -d

Step 2: Modify the way your production environment for the application under test runs to the following command

java -javaagent:/path/to/arex-agent-0.1.0.jar -Darex.service.name=your-service-name -Darex.storage.service.host=[storage.service.host:port](storage.service.host:port) -jar your-application.jar

Where：

• /path/to/arex-agent-0.1.0.jar needs to be changed to the actual path of the arex-agent-0.1.0.jar file in your server
• your-service-name needs to be changed to the name of your application under test
• [storage.service.host:port] needs to be changed to the address and port number of the data storage service
• your-application.jar needs to be changed to the jar package name of the application

Or change the environment variables in the environment

export JAVA_OPTS="-javaagent:/path/to/arex-agent-0.1.0.jar -Darex.config.path=/path/to/arex.agent.conf"

Step 3: Visit the AREX UI, and in the Replay tab, select the service that you have just loaded the AREX Agent on (i.e. your-service-name above).

Step 4: Select Replay, enter the Target Host of your test environment and start executing all the recorded test cases.

After the replay is completed, a detailed replay report will be generated, where you can visually see the pass rate, API pass rate, and the details under each API.

You can select the API that failed to replay to see the comparison results of the replay failure cases under that API, and AREX has visualized the comparison results in the latest version to facilitate locating the problem.

There is an even simpler way to start the local mode without installing the various cumbersome service components of AREX, simply by using ArexCli:

git clone https://github.com/arextest/arex-agent-java.git 
cd arex-agent-java 
mvn clean install  
java -jar "./arex-cli-parent/arex-cli/target/arex-cli.jar"

After running, the following is shown:

The following commands are supported:

• ls: view all recorded cases.

  • [option: -o/--operation] view all cases under the specified operation

• replay: replay the recorded data and compare the differences.

  • [option: -n/--num] replay numbers, default the latest 10
  • [option: -r/--recordId] single replay available for local debug

• watch: View replay results and differences.

  • [option: -r/--replayId] replay id, multiple are separated by spaces

Note that in local mode, AREX uses H2 as local storage to save test data and no longer relies on the storage service. Also, you will not be able to use the AREX UI.

API automation testing

AREX's automated testing actually tests a complete business process by combining multiple API test cases in order and then running them in batches.

One of the major advantages of using AREX tools for API automation testing is that there is no need to manually write a large number of test cases, and the coverage can be easily met by a large number of real requests online. You can choose to save the recorded API test cases directly or by recording the API test cases you need forcedly.

Save the Recorded Cases

AREX can record the real requests, including the mock data, of the AREX Agent-equipped application on the web with a certain sampling frequency. If you need to crystallize some important test cases for later regression testing, just find the required test cases in the replay report, selest Save, and select the target directory to save the test cases as regular test cases (the difference lies in the arex-record-id marker in the message header).

Forced Record

If some important test cases are not recorded, then can be saved by manually forced recording.

In the AREX API testing interface, select the action.record icon, add Key: arex-force-record, Value: true to the request Header, and select Send to send the request. Select Save to save the recording case.

Test cases that are recorded or forced recorded by AREX will have an additional tab of Mock data compared to normal test cases, i.e. all the data that AREX mocks during the recording process (on the left side are the request messages of the main interface and external calls that AREX mocks, and on the right side are the corresponding response messages).

AREX now supports editing Mock data. For example, if you need to validate a new feature developed in your local environment, and you are not satisfied with the Mock data, you can manually modify it to meet the needs of the new feature.

Automation Testing Pipeline

CI/CD is the standard for test teams nowadays, and supporting access to CI/CD is a must. AREX supports releasing WEB Hook interface to be triggered by external systems (such as Jenkins) for execution.

Create a test task in AREX, create test cases and test environment, and make sure the test task can run properly in AREX.

2. Create a new Job in Jenkins to trigger the AREX test task.

3. Add an "Execute shell" step to the Job configuration in Jenkins to execute a CURL command that invokes AREX's Web Hook interface. the CURL command can be constructed in the following ways:

curl -H "Content-Type: application/json" -X POST -d '{"test_id": "test_task_id"}' https://arex-webhook-url

Among them, arex-webhook-url is the URL of AREX's Web Hook interface, which can be obtained when select Start replay to start the replay test.

4. Configure a Jenkins Job's triggers so that the Job can run automatically under certain conditions. For example, you can set it to trigger every time you commit code to the source code repository.

5. Run the Jenkins Job and check that the test job is triggered and that the test results are normal.

By following these steps, we can integrate AREX into Jenkins for automated testing and continuous integration.

Left Shift in API Testing

The traditional software testing process usually includes the steps of requirements review, writing test cases, preparing test scripts, executing tests after development and testing, submitting problems, and regression testing. The problem with this process is that testers are in a very passive state and often need to wait for developers to complete code development and formal testing before they can carry out testing work, thus affecting the whole development process.

Especially in the era of widely used development models such as agile development and continuous delivery, the iteration cycle of Internet products has become shorter, faster and more frequent, which also brings more time pressure on traditional software testing.

To solve this problem, test left shift becomes a more proactive testing strategy. The main idea of test-left is to give testers more initiative to find and fix problems earlier by testing in the early stages of code development, thus reducing the impact of errors and failures on later development work. This way, testers are not delaying go-live every time due to poor quality or high risk, while also ensuring the quality of the product online.

In software development, developers use AREX for testing before code check-in by triggering a CI process. The purpose of this is to identify and fix problems before the code is checked in, reducing the impact on later development work. If a bug is identified during the issue verification process, the developer will fix it. If the change is a normal result of a code change, the developer will mark the difference as ignored. The test team is able to see the nodes that are marked as ignored and their changes, and is thus informed of the impact surface of the code change in parallel. This allows developers to be more proactive in dealing with issues, while also improving the quality and stability of the software.