Upgrade Tests to Parallel Execution

[CarlChaoCarl]

Background

Currently, the tests of Java-tron are executed serially, which makes it take a long time to execute all of them. For a Macbook Pro with a 2.6 GHz six-core Intel Core i7 processor, and 16 GB 2667 MHz DDR4 memory, it takes 24 minutes to run all tests.

Technically speaking, parallel execution of the tests compared to serial execution will significantly reduce the total execution time, thus improving R&D efficiency. Meanwhile, the coverage of the tests should be maintained.

Rationale

To enable test parallelism in java-tron, you need to change the configuration of gradle.

Gradle supports parallel execution of tests, which can be achieved by using Gradle's parallel testing function.

Use the following properties to control how the test process is started Test task:

maxParallelForks — default: 1

For example, let’s configure parallel testing in the build.gradle file of gradle and configure the maxParallelForks attribute in the test task.

test {
    maxParallelForks = 4
}

After this configuration, Gradle will run 4 test tasks in parallel.

Refer to Gradle's configuration documentation here: Testing in Java & JVM projects

By setting this property to a value greater than 1, tests can be run in parallel. This may make the tests run faster, especially with a multi-core CPU. When using parallel test execution, make sure your tests are properly isolated from each other. Tests that interact with the file system are particularly prone to conflicts, causing intermittent test failures.

This can be done by using the org.gradle.test.worker property, which is unique for each parallel worker. You can use this for anything you want, but it is especially useful for filenames and other resource identifiers to prevent the single-test conflicts mentioned above.

System.getProperty("org.gradle.test.worker")

Implementation

1. Enable test parallelism

To enable test parallelism in Java-tron, you need to make the following Gradle configuration changes

• a. In framework/build.gradle, add the following under the test module

test {
    if (System.getenv("CI") == null) {
        maxParallelForks = Runtime.runtime.availableProcessors().intdiv(2) ?: 1
    }
}

• b. In build.gradle, add the following under the subprojects module

subprojects {
    tasks.withType(JavaCompile).configureEach{
        options.fork = true
    }
}

2. Emerging Problems

The tests of Java-tron have incurred some new problems under parallel execution. Currently, the tests with these problems have been sorted out.

project	class	method
framework	DBConvertTest.java	testRun
plugins	DbLiteTest.java	testTools
framework	JsonrpcServiceTest.java	testGetBlockByNumber2
framework	SendCoinShieldTest.java	TestCreateMultipleTxAtTheSameTime
framework	SnapshotImplTest.java	All
framework	SnapshotRootTest.java	All

When these problematic tests are commented out, everything is fine when executing tests in parallel.

3. Solution

After changing to parallel execution, some tests occasionally failed to execute.These tests need to be refactored so that all tests of Java-tron can be executed in parallel.

4. Experiment of Parallel Execution

Experiments were conducted on an AMD EPYC 7R32 server with 64G memory and 32 core processors, and the following data were obtained:

enable test parallelism	parallel number	execution time	reduce percentage
no	1	about 18min	0%
yes	32	about 7min	about 60%
yes	16	about 7min	about 60%
yes	8	about 9min	about 50%

Based on the above experiments, enabling test parallelism on a multi-core machine can indeed reduce the execution time of tests, and the test parallelism can be set to = cpu core num / 2.

5. What to do in the future

After enabling parallel unit tests, the newly added unit tests need to be executed normally under parallel execution.

[Bellgin]

Seems to be a good time saver for the devs. How is the coverage of the tests?

[CarlChaoCarl]

@Bellgin

Yes, based on our experiments, after this issue is implemented, it will save developers time and thereby improve R&D efficiency.
Here, when tests are executed in parallel, the test coverage does not decrease at the same time.
At the same time, we have other issues to improve test coverage. Reference: #5434

[forfreeday]

This is a good improvement, with parallel execution turned on, do the test case methods suffer from each other.

[lurais]

@CarlChaoCarl Have you compared the test results after serial execution and parallel execution? Are the two guaranteed to be completely consistent? If parallel execution is enabled, is the order of single test execution actually completely random? And is the time of execution remains stable ？

[xxo1shine]

@CarlChaoCarl It is a good idea to execute tests in parallel to reduce running time. We look forward to implementing it. In addition, many tests need to start the system, which takes up a lot of time. Can the tests that need to start the system be merged?

[CarlChaoCarl]

This is a good improvement, with parallel execution turned on, do the test case methods suffer from each other.

Yes.
After turning on parallel execution, test cases will indeed affect each other.
So it must be optimized

@forfreeday

[CarlChaoCarl]

@CarlChaoCarl Have you compared the test results after serial execution and parallel execution? Are the two guaranteed to be completely consistent? If parallel execution is enabled, is the order of single test execution actually completely random? And is the time of execution remains stable ？

1. The test results after serial execution and parallel execution may be different, and the test writing method needs to be further reviewed and refactored.
2. Theoretically speaking, the execution results of the two must be completely consistent.
3. Enable parallel execution. The order of tests execution is random, and parallel execution in different parallel tasks is allocated.
4. After many experiments, the execution time after opening parallel execution is relatively stable.

@lurais

[CarlChaoCarl]

@CarlChaoCarl It is a good idea to execute tests in parallel to reduce running time. We look forward to implementing it. In addition, many tests need to start the system, which takes up a lot of time. Can the tests that need to start the system be merged?

Starting the system in test method is not a good way
You can use the indirect call system interface method or mock method to implement.

@ss3344520

[barbatos2011]

@CarlChaoCarl Hi, any future plan for this function？ Looking forward to the progress of the project~

[CarlChaoCarl]

@barbatos2011

Features are already under development.
Mainly refactor problematic tests and upgrade tests to parallel execution.
Related PR will be released later.
Thanks for attention.

[CarlChaoCarl]

Test parallelization is currently paused, so I temporarily close this issue.