Suite throws NoTestsDiscoveredException when all tests have been filtered

[hblohm]

We implemented a test engine (z2 Jupiter) that allows us to in-container tests. It's been in use for quite some time now and is useful only in context with the associated runtime environment (z2 - hence the name). It has been working nicely for Junit 4 and Junit 5 - but we are having trouble supporting @suite. Here is why - and it would be great to understand, if my thinking is wrong, or if Suite could be more friendly.

The approach is basically like this:

• During discovery, the client side of the engine sends the discovery request to the backend (container)
• On the backend, it runs local discovery and returns the result
• At the end of the client side discovery, in a post discovery filter, it filters out all tests discovered by other engines if it found them on the backend.
• At test execution, it delegates the actual execution to the backend and reports everything back so, as if tests were run on the client.

No, this kind of also works with @suite in so far as tests are executed remotely but SuiteTestDescriptor in the end throws an exception because non of the tests if discovered (before our post discovery filter) were actually run locally (because we took over).

Any suggestion on how to get around that would be great!

Thanks, Henning

[marcphilipp]

Are you referring to this exception being thrown?

https://github.com/junit-team/junit5/blob/6593317c15fb556febbde11914fa7afe00abf8cd/junit-platform-suite-engine/src/main/java/org/junit/platform/suite/engine/NoTestsDiscoveredException.java#L19-L21

If so, you can disable it by annotating your suite classes with @Suite(failIfNoTests = false). Does that help?

[hblohm]

Thank you for the fast reply! I will give it a try right away.

[hblohm]

Yes, that made it work. Should have seen that myself. Thanks a lot for your help

[mpkorstanje]

For failIfNoTests it might be worth differentiating between a lack of tests due to none being discovered and there being no tests due to pruning.

The goal was to help detect misconfiguration of the suite Engine. This would cover the former case, the latter would presumably be intentional.

And given that withRetainedEngines is used there should be enough information to make that distinction.

https://github.com/junit-team/junit5/blob/79c834ca504d90358ef99e9abe0bd74f145e994e/junit-platform-suite-engine/src/main/java/org/junit/platform/suite/engine/SuiteTestDescriptor.java#L166

[hblohm]

Can you elaborate a bit? I think I understand the intention of failIfNoTests. And yes, my case would be the intentional pruning. But is it possible to express that? As locally it is not "my" launcher that is running the discovery but, say, the IDE's launcher - is there a "more friendly" way than in post-discovery to express that locally only one engine should run?

[mpkorstanje]

The Suite Engine is a test Engine that uses the Launcher API. So it is effectively JUnit recursively executing JUnit.

Because it uses the Launcher API it has its internal discovery request and associated discovery result. It presents this plan externally by appending the descriptors from that plan to itself.

On https://github.com/junit-team/junit5/blob/79c834ca504d90358ef99e9abe0bd74f145e994e/junit-platform-suite-engine/src/main/java/org/junit/platform/suite/engine/SuiteTestDescriptor.java#L163-L168 it reconciles the external representation with the internal discovery request after pruning.

That this reconciliation happens shows it is possible to detect the difference between a result that was pruned to be empty and a result that was empty from the start.

I'm not sure how to use that information and if it is sufficient. You may want to investigate that.

is there a "more friendly" way than in post-discovery to express that locally only one engine should run?

The Launcher API offers the ability to request only tests from certain engines. But IDE support for the parts of JUnit 5 that didn't also exist in JUnit 4 is limited (#2849).

Though in your usecase you create the impression that you do still want all tests to run that weren't also discovered by z2 Jupiter. For that I'm not sure it is possible to use anything but pruning.

[hblohm]

The mechanism for z2 Jupiter is another variation of the same mechanism: Locally it removes all test cases except its own - which are all test cases discovered remotely (but now, client side, owned by the z2 Jupiter engine). Remotely it uses the launcher API to run all tests discovered by other engines but itself - but then tunnels the listener events to the client (that consequently claims them as being its own).

Anyway, requiring the failIfNoTests==false is a workaround I can perfectly live with. Thanks!