Running @QuarkusTest in Parallel

[bcluap]

Is it possible to split ones tests across multiple classes and run them in parallel against one instance of Quarkus?

I've managed to get JUnit ordering working and can run test classes in an order I want, but many of the tests take a long time (they are integration tests and call into Quarkus that then calls into external APIs). So lets say I have 10 test classes annotated with @QuarkusTest - and each one takes say 2min to run. Right now they run one after the other and takes 20 minutes. Can I possibly run all 10 in parallel and hence taking more like 2min (the tests are not CPU bound).

If I use this config:
junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.classes.default=concurrent
junit.jupiter.execution.parallel.mode.default=same_thread

Then JUnit tries to start up multiple instances of Quarkus as opposed to using once instance and running the tests in parallel.

[turf00]

I also encountered the same problem and was wondering what the general suggested solution is.

[roggenbrot]

Since I'm trying to implement parallelism for the quarkus-spock extension at the moment I would be interested in the answer as well.

Looking at the source code of the QuarkusTestExtension the quarkus application is a static variable

quarkus/test-framework/junit5/src/main/java/io/quarkus/test/junit/QuarkusTestExtension.java

Line 125 in e47a267

private static RunningQuarkusApplication runningQuarkusApplication;

so as far as the test execution is not forked (not a junit5 expert... ) or a custom classloading mechanism is used for each test class (seems not to be supported in junit 5 junit-team/junit5#201) my guess is that running quarkus test specifications in parallel is not supported at the moment.

Could someone confirm/deny this guess?

[roggenbrot]

What you can do is enabling forking in surefire:

<plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>${surefire-plugin.version}</version>
      <configuration>
          <forkCount>4</forkCount>
          <reuseForks>false</reuseForks>
          <systemPropertyVariables>
              <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
              <maven.home>${maven.home}</maven.home>
          </systemPropertyVariables>
      </configuration>
  </plugin>

Make sure that you've set you ports config to dynamic ports (e.g. quarkus.http.test-port=0)

[bcluap]

I did a workaround that solves for my use case. It allows one to have many test classes running in parallel and all test against the same running instance of Quarkus. Our tests went from 15 minutes to 1 minute. No need to have lots of Quarkus instances running. The steps are a bit hackey but work perfectly. The just of it is that Quarkus kicks off a test that in turn uses the JUnit libraries to kick off your actual tests in parallel:

1. junit-platform.properties in src/test/resources contains something along the lines of:

junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.config.fixed.parallelism=5
junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.mode.classes.default=concurrent
junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.testclass.order.default=org.junit.jupiter.api.ClassOrderer$OrderAnnotation
junit.jupiter.testmethod.order.default=org.junit.jupiter.api.MethodOrderer$OrderAnnotation

2. Have a single "driver" test class annotated with @QuarkusTest This class is all Quarkus sees as the integration test but it in turn kicks off the actual tests. Here is my source:

@QuarkusTest
public class DriverTest {

    private static final Logger log = LoggerFactory.getLogger(EclipseTest.class);
    private boolean testsFinished = false;

   

    @Test
    @Order(1)
    void startup() {
        log.info("Starting Quarkus to run integration tests");
        // Do something to know Quarkus has started. You could just sleep if you want
        log.info("Quarkus has started. Now running tests");
        int failures = runAllTests();
        assertThat(failures).isZero();
    }

    public int runAllTests() {
        SummaryGeneratingListener summaryListener = new SummaryGeneratingListener();
        String filter = ".*";
        log.info("Running all tests with class name matching [{}]", filter);
        LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request()
                .selectors(selectPackage("com.ukheshe.eclipse.rest.tests"))
                .filters(includeClassNamePatterns(filter))
                .build();
        Launcher launcher = LauncherFactory.create();
        launcher.execute(request, summaryListener, new TestExecutionListener() {
            @Override
            public void testPlanExecutionFinished(TestPlan testPlan) {
                log.info("Test plan is finished");
                testsFinished = true;
            }

            @Override
            public void testPlanExecutionStarted(TestPlan testPlan) {
                log.info("\n\n\n######################################################\n"
                        + "We will be running [{}] tests\n"
                        + "######################################################\n\n\n",
                        summaryListener.getSummary().getTestsFoundCount());
            }
        });
        while (!testsFinished) {
            Thread.currentThread.sleep(1000);
        }
        System.out.println("######################################################");
        summaryListener.getSummary().printFailuresTo(new PrintWriter(System.out), 0);
        summaryListener.getSummary().printTo(new PrintWriter(System.out));
        System.out.println("######################################################");

        
        return summaryListener.getSummary().getFailures().size();
    }
}

3. Your actual unit test classes can just be annotated with normal Junit @test annotation e.g.

public class TestFunctionalArea1  {
    @Test
    void someTest() {
...

[mkouba]

Is it possible to split ones tests across multiple classes and run them in parallel against one instance of Quarkus?

I don't think that this is officialy supported and could also cause troubles when using testing profiles.

But @geoand @aloubyansky @gsmet and others will know more for sure ;-)

[geoand]

Currently, it will not work properly

[franden]

we have the same requirement for parallel execution of @QuarkusTest test classes. is there any chance that this will be supported in near future?

[ymajoros]

Any news on this?

[geoand]

Nope

[bcluap]

@geoand and @holly-cummins Quarkus 3.22 has unfortunately broken this workaround as the classloader changes seem to mess with this.

15:30:19,589: [main][com.ukheshe.eclipse.rest.DriverTest:startup][ERROR]{sampled=false, spanId=0000000000000000, traceId=00000000000000000000000000000000}Error occured running tests:: java.lang.UnsupportedOperationException: Using io.quarkus.logging.Log is only possible with Quarkus bytecode transformation; make sure the archive is indexed, for example by including a beans.xml file
	at io.quarkus.logging.Log.fail(Log.java:2927)
	at io.quarkus.logging.Log.debugf(Log.java:898)
	at io.quarkus.test.junit.classloading.FacadeClassLoader.loadClass(FacadeClassLoader.java:232)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:543)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1088)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1099)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1142)
	at java.base/java.util.ServiceLoader$1.hasNext(ServiceLoader.java:1164)
	at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1246)
        at java.base/java.lang.Iterable.forEach(Iterable.java:74)
	at org.junit.platform.launcher.core.LauncherFactory.collectTestEngines(LauncherFactory.java:158)
	at org.junit.platform.launcher.core.LauncherFactory.createDefaultLauncher(LauncherFactory.java:134)
	at org.junit.platform.launcher.core.LauncherFactory.lambda$create$2(LauncherFactory.java:128)
	at org.junit.platform.launcher.core.ClasspathAlignmentCheckingLauncherInterceptor.intercept(ClasspathAlignmentCheckingLauncherInterceptor.java:25)
	at org.junit.platform.launcher.core.DefaultLauncherSession$2.lambda$intercept$0(DefaultLauncherSession.java:133)
	at io.quarkus.test.junit.launcher.CustomLauncherInterceptor.actuallyIntercept(CustomLauncherInterceptor.java:50)
	at io.quarkus.test.junit.launcher.CustomLauncherInterceptor.intercept(CustomLauncherInterceptor.java:22)
	at org.junit.platform.launcher.core.DefaultLauncherSession$2.intercept(DefaultLauncherSession.java:133)
	at org.junit.platform.launcher.core.DefaultLauncherSession.<init>(DefaultLauncherSession.java:57)
	at org.junit.platform.launcher.core.SessionPerRequestLauncher.createSession(SessionPerRequestLauncher.java:75)
	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:62)
	at com.ukheshe.eclipse.rest.DriverTest.runAllTests(DriverTest.java:115)...

Any thoughts on a workaround to support running junit tests within QuarkusTest so that we have ability to run tests in parallel. Running tests one after the other is a showstopper for us. We have over 1400 tests and most take a few seconds at least as they do blocking calls to services and things.

[geoand]

What was your workaround doing?

[bcluap]

As per #29218 (comment)

[holly-cummins]

Going forward (as in future feature development), it seems like it would be not-unreasonable to have tests in the same profile able to run in parallel, because there wouldn't be port conflicts. The new class orderer could maybe do it, or maybe test profiles could come with a built-in @ResourceLock, or ... something. ExecutionListener would also need to be fixed.

But that would be a future thing. In the short term, the stack trace you're getting almost looks like a simple code error in how FacadeClassLoader is doing its debugging, except I've seen that code working.

Have you got logging enabled? If so, does it work to just disable logging? I assume it's not going to be that easy ... :)

[bcluap]

Thanks @holly-cummins . I'm not sure if this is an issue:

It seems like its using a static method on a Log object in some places instead of the class instance logger

[holly-cummins]

Thanks, @bcluap. That line you've highlighted is definitely the line in the stack trace. There's no reason to do it the lazy way with the static method, so I'll fix it for 3.23. What I don't know is (a) why that line is causing problems in your scenario and not elsewhere and (b) what error you'll hit next :)

[holly-cummins]

PR: #47822

[bcluap]

Thanks so much @holly-cummins . I used that PR and it all works now :-)

[holly-cummins]

Thanks for letting me know!