Test Suite Design with 100% Mutation Score

The source code and especially the test suite provides an example with 100% mutation score.

Mocks vs TestHarness

One of the design goals of the test suite was to avoid the use of mocking. The testsuite makes heavy use of test harnesses provided by the SonarQube test API as well as custom test harnesses.

However there was one case - the ReportCollectorTest - where the use of mocking was inevitable to induce an IOException on filesystem level. For all other tests we accomplished to arrange the test setups using standard APIs of SonarQube, Java and test harnesses.

Effects on the code base

The code base also provides some example of the (negative) effects of 100% mutation score. In other words, a "killing spree", aiming at 100% mutation score, might end up in less good code. For this plugin we did it on purpose for demonstration effects and the cases were rather rare. In the following sections we'd like to share our insights.

Removing optimizations

In some situations, short-cut code had to be removed because the existence of it was an equivalent mutation. For instance a check list.isEmpty() is unnecessary if later on a stream iterates on zero elements.

if(list.isEmpty()) { //removing this short-cut is an equivalent mutation
  return;
}
list.stream()....

In "normal" such a removal - especially when it actually provides a performance optimization - is not recommendable, because it may prevent a much more expensive operation, such as creating a stream.

Bloating code

While writing the tests, there were two cases where changing the code to kill otherwise un-killable mutants was inevitable: varargs and enums with constructors.

Varargs

The uses of varargs (i.e. method(String... args)) produces compiler-generated artifacts which Pitest (at least in version 1.4) does not recognize as such and mutates them. But there are no means for killing them.

One thing Pitest does, is reordering the elements of the varargs. In case the order does not matter, i.e. when adding Extension classes to the Plugin Context, any reordering is an equivalent mutation.

Overspecification

One way to kill this mutant is to over-specify by testing for an order.

For example, for this code:

context.addExtensions(Class1.class, Class2.class, Class3.class);

The InlineConst-Mutator generates mutations like "Substituted 1->0", "Substituted 2->3" etc

With a like this

List<Class> extensions = context.getExtensions();
assertEquals(Class1.class, extensions.get(0));
assertEquals(Class2.class, extensions.get(0));
assertEquals(Class3.class, extensions.get(0));

These could be killed. However a big disadvantage is, that this specification is not needed as the order does not matter. Another effect is, that does not even kill all mutants. For some reasons, an additional mutation is generated that mutates a constant beyond the size of the array (for varargs[3] something like "Substituted 3->4" is produced) which is not killable - an "Off-by-one Super-Mutant" - true nightmare in computer science (naming wise and off-by-one wise :).

Alternative Methods

The option is to search for alternate methods to use. For example the SonarQube plugin API provides a way to add Extensions class by class, see MutationAnalysisPlugin.

Instead of

context.addExtensions(Class1.class, Class2.class, Class3.class);

Write

context.addExtension(Class1.class);
context.addExtension(Class2.class);
context.addExtension(Class3.class);

and test it with

assertTrue(context.getExtensions().contains(Class1.class));

Helper Methods

Another "trick" we applied is use utility methods that produce an array without using compiler generated inline consts.

For the sensors, the languages and rules repositories have to be specified using a vararg:

descriptor.onlyOnLanguages("java", "kotlin");

Replacing this with a an array onlyOnLanguages(new String[]{"java", "kotlin"}) doesn't the kill the Off-by-one Super Mutant ("Substituted 2->3").

So we sought for a way to create the array without using varargs or array-initializers. Therefore we created a helper method toArray that accepted fixed number of argument, creates a list from it and used the Stream API to stream/collect the list into an array.

private String[] toArray(String element1, String element2) {
    final List<String> list = new ArrayList<>();
    list.add(element1);
    list.add(element2);
    return list.stream().toArray(String[]::new);
}

Calling toArray(new String[0]) directly on the list would introduce another mutation on the 1 argument, which again is equivalent as the toArray only uses the array to the get the type of the array.

An example of this can be found in PitestSensor

Enums with Constructors

Extraneous Code & Over-SpecificationIn on

StreamSupport

VarArgs order