Skip to content

[AUTOCUT] Gradle Check Flaky Test Report for SmokeTestMultiNodeClientYamlTestSuiteIT #14408

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
opensearch-ci-bot opened this issue Jun 17, 2024 · 2 comments
Open

[AUTOCUT] Gradle Check Flaky Test Report for SmokeTestMultiNodeClientYamlTestSuiteIT #14408

opensearch-ci-bot opened this issue Jun 17, 2024 · 2 comments
Assignees
@sandeshkr419
Labels
autocut flaky-test Random test failure that succeeds on second run Search:Aggregations Search Search query, autocomplete ...etc >test-failure Test failure from CI, local build, etc.

Comments

@opensearch-ci-bot
Copy link
Collaborator

opensearch-ci-bot commented Jun 17, 2024 •
edited
Loading

Flaky Test Report for SmokeTestMultiNodeClientYamlTestSuiteIT

Noticed the SmokeTestMultiNodeClientYamlTestSuiteIT has some flaky, failing tests that failed during post-merge actions.

Details

Git Reference Merged Pull Request Build Details Test Name
1015fa6 15154 44203 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=indices.delete_index_template/10_basic/Delete index template which is not used by data stream but index pattern matches}
1e9fdb4 15410 45949 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=index/91_flat_object_null_value/Supported queries}
481fd38 16098 48529 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.classMethod

org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search/340_doc_values_field/search on fields with both index and doc_values enabled}
017f7d4 15704 47021 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/360_date_histogram/Date histogram aggregation w/ shared field range test}
07029b2 15906 47796 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/360_date_histogram/Date histogram aggregation w/ shared field range test}
10b955d 15547 46008 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/360_date_histogram/Date histogram aggregation w/ shared field range test}
1b48dbd 18090 57280 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/90_sig_text_field_match_only_text/Dedup noise}
4a48b4b 17147 52716 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/20_terms/string profiler via global ordinals}
61dbcd0 14348 41143 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/230_composite/composite aggregation date_histogram profile shows filter rewrite info}
81288b1 15957 47925 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/360_date_histogram/Date histogram aggregation w/ shared field range test}
a24c858 15805 47501 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/360_date_histogram/Date histogram aggregation w/ shared field range test}
af25ece 15969 48106 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/360_date_histogram/Date histogram aggregation w/ shared field range test}
b842e48 17902 56242 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/20_terms/string profiler via global ordinals}
e076e66 15348 44994 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/360_date_histogram/Date histogram aggregation w/ shared field range test}

The other pull requests, besides those involved in post-merge actions, that contain failing tests with the SmokeTestMultiNodeClientYamlTestSuiteIT class are:

For more details on the failed tests refer to OpenSearch Gradle Check Metrics dashboard.
@opensearch-ci-bot opensearch-ci-bot added >test-failure Test failure from CI, local build, etc. autocut flaky-test Random test failure that succeeds on second run untriaged labels Jun 17, 2024
@peternied peternied added Search:Aggregations Search Search query, autocomplete ...etc labels Jun 19, 2024
@reta reta mentioned this issue Jun 19, 2024
Closed
@getsaurabh02 getsaurabh02 moved this from 🆕 New to Later (6 months plus) in Search Project Board Aug 15, 2024
@ashking94 ashking94 mentioned this issue Aug 28, 2024
Merged
2 tasks
@ltaragi ltaragi mentioned this issue Sep 3, 2024
Open
3 tasks
@finnegancarroll finnegancarroll mentioned this issue Sep 24, 2024
Merged
3 tasks
@prudhvigodithi prudhvigodithi mentioned this issue Nov 13, 2024
Open
@sandeshkr419 sandeshkr419 self-assigned this Dec 11, 2024
@sandeshkr419
Copy link
Contributor

sandeshkr419 commented Jan 6, 2025 •
edited
Loading

#16067 has fixed tests related to 360_date_histogram

Last reported failures were in #16421 and #16254:

This test detail is not covered in list of tests in issue description, and with last occurrence on 10/25/2024

java.lang.AssertionError: Failure at [search.aggregation/20_terms:1082]: profile.shards.0.aggregations.0.debug.segments_with_multi_valued_ords didn't match expected value:
profile.shards.0.aggregations.0.debug.segments_with_multi_valued_ords: expected Integer [0] but was Integer [1]

REPRODUCE WITH: ./gradlew ':qa:smoke-test-multinode:integTest' --tests "org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT" -Dtests.method="test {yaml=search.aggregation/20_terms/string profiler via global ordinals}" -Dtests.seed=5547B5C90D78DE0F -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ja-Jpan-JP -Dtests.timezone=Mexico/BajaSur -Druntime.java=21

REPRODUCE WITH: ./gradlew ':qa:smoke-test-multinode:integTest' --tests "org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT" -Dtests.method="test {yaml=search.aggregation/20_terms/string profiler via global ordinals}" -Dtests.seed=47181BE7B412E53 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sd-PK -Dtests.timezone=America/Guayaquil -Druntime.java=23

Tried reproducing using the same seeds as above, but not able to reproduce.

Build details not retained for previous builds, meaning tests in description haven't failed since Oct 9. Checking further if we fixed those tests indirectly.

Will wait and observe if there are new failures in couple days, else will close it out since the bot will recreate the issue with fresh details or reopen it.

@andrross
Copy link
Member

@sandeshkr419 Looks like this has been failing again recently. Here's a case that just happened:

REPRODUCE WITH: ./gradlew ':qa:smoke-test-multinode:integTest' --tests "org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT" -Dtests.method="test {yaml=search.aggregation/20_terms/string profiler via global ordinals}" -Dtests.seed=8167B4B83B03FD7D -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=gsw-FR -Dtests.timezone=Asia/Qyzylorda -Druntime.java=21

SmokeTestMultiNodeClientYamlTestSuiteIT > test {yaml=search.aggregation/20_terms/string profiler via global ordinals} FAILED
    java.lang.AssertionError: Failure at [search.aggregation/20_terms:1078]: profile.shards.0.aggregations.0.debug.segments_with_multi_valued_ords didn't match expected value:
    profile.shards.0.aggregations.0.debug.segments_with_multi_valued_ords: expected Integer [0] but was Integer [1]
        at __randomizedtesting.SeedInfo.seed([8167B4B83B03FD7D:9338B6295FF9085]:0)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:460)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.test(OpenSearchClientYamlSuiteTestCase.java:433)
        at java.****/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.****/java.lang.reflect.Method.invoke(Method.java:580)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.****/java.lang.Thread.run(Thread.java:1583)

        Caused by:
        java.lang.AssertionError: profile.shards.0.aggregations.0.debug.segments_with_multi_valued_ords didn't match expected value:
        profile.shards.0.aggregations.0.debug.segments_with_multi_valued_ords: expected Integer [0] but was Integer [1]
            at org.opensearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:115)
            at org.opensearch.test.rest.yaml.section.Assertion.execute(Assertion.java:126)
            at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:449)
            ... 39 more

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sandeshkr419 sandeshkr419

Labels
autocut flaky-test Random test failure that succeeds on second run Search:Aggregations Search Search query, autocomplete ...etc >test-failure Test failure from CI, local build, etc.
Projects
Search Project Board
Status: Later (6 months plus)
Milestone
No milestone
Development

No branches or pull requests
4 participants
@peternied @andrross @sandeshkr419 @opensearch-ci-bot