[BUG] Flaky test failure - v1.2.5#mixedClusterTest - yaml rest test - indices.get_field_mapping

[tlfeng]

Describe the bug
Seen in #2437 (comment)

Gradle run result:
Log 3252
Reports 3252

To Reproduce
I can't reproduce it locally. 🙂
The below shows in the log of Gradle check.

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v1.2.5#mixedClusterTest' --tests "org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=indices.get_field_mapping/20_missing_field/Return empty object if field doesn't exist, but index does}" -Dtests.seed=9FC2465A92C8B157 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=uk-UA -Dtests.timezone=Australia/Hobart -Druntime.java=17

org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=indices.get_field_mapping/20_missing_field/Return empty object if field doesn't exist, but index does} FAILED
    java.lang.AssertionError: Failure at [indices.get_field_mapping/20_missing_field:13]: expected [2xx] status code but api [indices.get_field_mapping] returned [404 Not Found] [{}]
        at __randomizedtesting.SeedInfo.seed([9FC2465A92C8B157:179679803C34DCAF]:0)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:442)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.test(OpenSearchClientYamlSuiteTestCase.java:415)

Expected behavior
test passes.

Plugins
none.

Screenshots
none.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Occurred since Mar.10th, 2022.

The BWC test running with OpenSearch v1.3.0 can also fail.
gradle command: ./gradlew ':qa:mixed-cluster:v1.3.0#mixedClusterTest' --tests "org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=indices.get_field_mapping/20_missing_field/Return empty object if field doesn't exist, but index does}" -Dtests.seed=2E22A45097655A52
see #2438 (comment)

[tlfeng]

There is an attempt to fix in the commit 4577e73

The test failure message became to the below after the above commit:

> Task :qa:mixed-cluster:v1.2.5#mixedClusterTest

REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v1.2.5#mixedClusterTest' --tests "org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=indices.get_field_mapping/20_missing_field/Return empty object if field doesn't exist, but index does}" -Dtests.seed=3BDEACA9E1F4F10C -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-PR -Dtests.timezone=Asia/Baghdad -Druntime.java=17

org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=indices.get_field_mapping/20_missing_field/Return empty object if field doesn't exist, but index does} FAILED
    java.lang.AssertionError: Failure at [indices.get_field_mapping/20_missing_field:20]: field [test_index.mappings] is null
        at __randomizedtesting.SeedInfo.seed([3BDEACA9E1F4F10C:B38A93734F089CF4]:0)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:442)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.test(OpenSearchClientYamlSuiteTestCase.java:415)

See #2439 (comment)

❌   Gradle Check failure 4577e73
Log 3273
Reports 3273

[tlfeng]

I tested the REST API manually in OpenSearch single node of both 1.x branch and 2.0 branch:

PUT http://localhost:9200/test_index

{
    "mappings": {
      "properties": {
        "text": {
          "type": "text",
          "analyzer": "whitespace"
        }
      }
    }
}

GET http://localhost:9200/test_index/_mapping/field/not_existent

Unfortunately, I didn't get 404 response code, or empty response body {}. Maybe it will only shows error in cluster with nodes of both 1.x and 2.0 versions.