StubRepository treats every file as a potential mapping, leading to flaky tests

[daniel-frak]

Describe the bug

In my project I'm using the spring-cloud-dependencies version 2024.0.1, however, the issue is reproducible using Spring Cloud Contract samples, as well.

The problem is that StubRepository::collectMappings walks the entire contract folder tree and (given that stubs per consumer is off), de facto only checks if a file ends in .json and is resolvable to com.github.tomakehurst.wiremock.stubbing.StubMapping (WireMockHttpServerStub::isAccepted).

This leads to a situation where a simple JSON bodyFromFile file can, for example, be mistakenly recognized as a mapping, creating a "catch-all" mapping which can override other, more specific ones. Given that the order of these mappings is not guaranteed, this can lead to flaky consumer tests.

Sample - Spring Cloud Contract

The limited StubRepository mapping checks pose a problem when using bodyFromFile in contracts, like in the case of Spring Cloud Contract samples, where src/test/resources/contracts/beer/rest/response.json is used in src/test/resources/contracts/beer/rest/shouldGrantABeerIfOldEnoughFromFile.yml.

When src/test/java/com/example/BeerControllerYamlTest.java is executed, both files are included in the generated /tmp/contracts-[...]/META-INF/com.example/beer-api-producer-yaml/0.0.1-SNAPSHOT/contracts folder, where StubRepository later evaluates response.json as a potential mapping:

response.json only misses becoming a mapping by a hair, due to status not being recognized:

Sample - a mistakenly recognized mapping

In a sample application of my own, which is not yet public, I have created the following contract test.

src/test/resources/contracts/http/test_spec.yml:

name: Should return test message
request:
  url: /test
  method: POST
  bodyFromFile: bodies/test_request_body.json
  headers:
    Content-Type: "application/json"
response:
  status: 200
  bodyFromFile: bodies/test_response_body.json
  headers:
    Content-Type: "application/json"

src/test/resources/contracts/http/bodies/test_request_body.json:

{
  "name": "World"
}

src/test/resources/contracts/http/bodies/test_response_body.json:

{
  "response": "Hello, World (from test producer at 2025-01-01T00:00:00Z)!"
}

Using @AutoConfigureStubRunner(mappingsOutputFolder = "target/outputmappings/", [...]), I get the following mappings:

[{
  "id" : "04f1fcf7-2d79-4e91-aa9e-9e2ab21ec1a3",
  "name" : "World",
  "request" : {
    "method" : "ANY"
  },
  "response" : {
    "status" : 200
  },
  "uuid" : "04f1fcf7-2d79-4e91-aa9e-9e2ab21ec1a3"
},
{
  "id" : "ebc34b86-c49c-42b7-ad25-e7dd35a11f73",
  "request" : {
    "urlPath" : "/test",
    "method" : "POST",
    "headers" : {
      "Content-Type" : {
        "equalTo" : "application/json"
      }
    },
    "bodyPatterns" : [ {
      "matchesJsonPath" : "$[?(@.['name'] == 'World')]"
    } ]
  },
  "response" : {
    "status" : 200,
    "body" : "{\"response\":\"Hello, World (from test producer at 2025-01-01T00:00:00Z)!\"}",
    "headers" : {
      "Content-Type" : "application/json"
    },
    "transformers" : [ "response-template", "spring-cloud-contract" ]
  },
  "uuid" : "ebc34b86-c49c-42b7-ad25-e7dd35a11f73"
},
{
  "id" : "23c9aaf5-3c1a-43b7-a581-7c462b7e279a",
  "request" : {
    "url" : "/health",
    "method" : "GET"
  },
  "response" : {
    "status" : 200,
    "body" : "OK"
  },
  "uuid" : "23c9aaf5-3c1a-43b7-a581-7c462b7e279a"
},
{
  "id" : "1c0088fd-7ebb-4314-923f-d35d7713d15c",
  "request" : {
    "url" : "/ping",
    "method" : "GET"
  },
  "response" : {
    "status" : 200,
    "body" : "OK"
  },
  "uuid" : "1c0088fd-7ebb-4314-923f-d35d7713d15c"
}]

As you can see, a catch-all mapping called "World" was created, because test_request_body.json is just similar enough to StubMapping to be successfully deserialized as such. This catch-all mapping sometimes gets put first in the order, leading to my consumer test failing, and sometimes it gets generated later, leading to the test succeeding.

Sample - a malicious file

In fact, I believe destroying any test pipeline is as simple as adding a simple .json file anywhere within the contracts folder, like:

{
  "name": "OH MY"
}

Whether or not this file is used as bodyFromFile, it will create a "catch-all" mapping which will make contract tests flaky - even though I'm only using yml files for my contract definitions!

Conclusion

There must be a better mechanism in StubRepository for recognizing bodyFromFile inclusions as different from mappings, as currently it's mostly based on luck.

I can make my sample app public, if needed, but I think there's enough in this description to reproduce the problem. Let me know if I can provide any more information to help with this issue.