No tests found from failsafe with newer JUnit5 and Cucumber

[madduci]

I have upgraded my Test project to the latest version of libraries and the Cucumber Tests get discovered, but the Failsafe Plugin reports that it has executed 0 tests, as same as for the Serenity Plugin.

Environment

Java: 21.0.6
Maven: 3.9.9
Serenity: 4.2.23
Cucumber: 7.22.2
JUnit Jupiter: 5.12.2
JUnit Platform Suite: 1.12.2

TestRunner Class

The TestRunner Class registered in Maven Failsafe is so implemented:

package de.gematik.test.demis.notificationPortal;

import io.cucumber.core.options.Constants;
import org.junit.platform.suite.api.ConfigurationParameter;
import org.junit.platform.suite.api.IncludeEngines;
import org.junit.platform.suite.api.SelectClasspathResource;
import org.junit.platform.suite.api.SelectPackages;
import org.junit.platform.suite.api.Suite;
import org.junit.platform.suite.api.SuiteDisplayName;

@Suite
@SuiteDisplayName("Notification Portal Test Suite")
@IncludeEngines("cucumber")
@SelectPackages("de.gematik.test.demis.notificationPortal")
@SelectClasspathResource("/features")
@ConfigurationParameter(
    key = Constants.GLUE_PROPERTY_NAME,
    value =   "de.gematik.test.demis.notificationPortal.hooks,net.serenitybdd.cucumber.actors,de.gematik.test.demis.notificationPortal.glue")
@ConfigurationParameter(
    key = Constants.PLUGIN_PROPERTY_NAME,
    value = "pretty,timeline:test-output-thread/")
public class TestRunner {}

Expected Behaviour

Tests are recognized by Failsafe and the Serenity Plugins, the build breaks in case of errors.

Observed Behavior

Maven Failsafe Plugin starts the TestRunner class tagged as @Suite and executes the Cucumber features, but at the end reports that it has executed 0 tests. The build is always successful, also in case of errors.

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

The Serenity Maven Plugin reports this output too:

[INFO] --- serenity:4.2.23:aggregate (serenity-reports) @ demis-notification-portal-test ---
[INFO] GENERATING REPORTS FOR: /home/jenkins/agent/workspace/DEMIS-Testsuite_notification_portal-LINUX-RUN
[INFO] GENERATING REPORTS USING 14 THREADS
[INFO] GENERATING SUMMARY REPORTS...
[INFO] GENERATING REQUIREMENTS REPORTS...
[INFO] GENERATING RESULT REPORTS...
[INFO] GENERATING ERROR REPORTS...
[INFO] Test results for 0 tests generated in 2.3 secs in directory: file:/home/jenkins/agent/workspace/target/site/serenity/
[INFO]   - Full Report: file:///home/jenkins/agent/workspace//target/site/serenity/index.html

Funnily, the Report ist then still generated, contains also the real results from tests, but the build is marked as succesful

[madduci]

I have cloned this Project and upgraded the libraries to reflect my own ones and i could identify the problem that with the `maven-failsafe-plugin 3.5.3 the tests are not identified correctly, but with the 3.5.2 Version they work.

[wakaleo]

Could be a JUnit 5 dependency conflict

[madduci]

@wakaleo i have pinned the versions through <dependencyManagement> and the usage of cucumber and junit-bom.

[wakaleo]

I have cloned this Project and upgraded the libraries to reflect my own ones and i could identify the problem that with the `maven-failsafe-plugin 3.5.3 the tests are not identified correctly, but with the 3.5.2 Version they work.

So maybe a defect in the failsafe plugin you think?

[madduci]

it seems like a big regression, i've found also some open Issues here -

apache/maven-surefire#834
apache/maven-surefire#835
apache/maven-surefire#837

[wakaleo]

Thanks for confirming - I tried with the latest snapshot and latest JUnit and Cucumber versions and can confirm that it appears to be related to maven-failsafe-plugin version 3.5.3.

A workaround appears to be to use surefire instead of failsafe.

[olamy]

should be fixed with apache/maven-surefire#828