[AUTOCUT] Gradle Check Flaky Test Report for SearchIT

[opensearch-ci-bot]

Flaky Test Report for SearchIT

Noticed the SearchIT has some flaky, failing tests that failed during post-merge actions.

Details

Git Reference	Merged Pull Request	Build Details	Test Name
22a6194	18124	58554	org.opensearch.client.SearchIT.testSearchWithMatrixStats
2c1b2aa	18247	57899	org.opensearch.client.SearchIT.testSearchWithMatrixStats
83a4beb	18073	57316	org.opensearch.client.SearchIT.testSearchWithMatrixStats

The other pull requests, besides those involved in post-merge actions, that contain failing tests with the SearchIT class are:

• 14443
• 15258
• 16294
• 17507
• 16421
• 18332
• 16607
• 17275
• 16366
• 16378
• 17457
• 17909
• 17611
• 17446
• 17817
• 18230
• 18194
• 16891
• 17961
• 18333
• 17538
• 17890
• 17907
• 17941
• 18043
• 18054
• 18069
• 18085
• 18102
• 18107
• 18155
• 18161
• 18201
• 18212
• 18243
• 18250
• 18327
• 18346
• 18351

For more details on the failed tests refer to OpenSearch Gradle Check Metrics dashboard.

[andrross]

REPRODUCE WITH: ./gradlew ':client:rest-high-level:asyncIntegTest' --tests "org.opensearch.client.SearchIT.testSearchWithMatrixStats" -Dtests.seed=630A15BDCB6BB5D9 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ksb -Dtests.timezone=Europe/Jersey -Druntime.java=21

SearchIT > testSearchWithMatrixStats FAILED
    java.lang.AssertionError: expected:<1830.0000000000002> but was:<1829.9999999999998>
        at __randomizedtesting.SeedInfo.seed([630A15BDCB6BB5D9:1301905AE5084B97]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:555)
        at org.junit.Assert.assertEquals(Assert.java:685)
        at org.opensearch.client.SearchIT.testSearchWithMatrixStats(SearchIT.java:570)

Looks like this test is expecting a very specific value here:

OpenSearch/client/rest-high-level/src/test/java/org/opensearch/client/SearchIT.java

Line 570 in 7c46f8f

assertEquals(1830.0000000000002, matrixStats.getVariance("num"), 0d);

[andrross]

#17978 could be the cause here. @bowenlan-amzn @sohami @Vikasht34 This is a newly flaky test. Would concurrent search by default explain the issue here?

[bowenlan-amzn]

@andrross I will look into this
We did see issue of small mismatch of matrx stats agg and add tolerance for that 4a92aa4

[dbwiddis]

I'm seeing this flaky a lot on my unrelated PR(s).

java.lang.AssertionError: expected:<1830.0000000000002> but was:<1830.0>

java.lang.AssertionError: expected:<1830.0000000000002> but was:<1830.0000000000005>

There are several tests which provide a 0d tolerance interval.

[kkewwei]

@bowenlan-amzn Could you mind have a look in your spare time?