Skip to content

Commit c0e1d49

Browse files
kaushalmahi12shiv0408alchemist51gaobinlongRS146BIJAY
authored and
wangdongyu.danny
committed
Add queryGroupId to search workload tasks at co-ordinator and data node level (opensearch-project#14708)
* add logic to add headers to Task Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add logic to add queryGroupId to task headers Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * remove redundant code Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add changelog entry Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix precommit Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add UTs for RemoteIndexMetadataManager (opensearch-project#14660) Signed-off-by: Shivansh Arora <hishiv@amazon.com> Co-authored-by: Arpit-Bandejiya <abandeji@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Fix match_phrase_prefix_query not working on text field with multiple values and index_prefixes (opensearch-project#10959) * Fix match_phrase_prefix_query not working on text field with multiple values and index_prefixes Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Add more test Signed-off-by: Gao Binlong <gbinlong@amazon.com> * modify change log Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Fix test failure Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Change the indexAnalyzer used by prefix field Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Skip old version for yaml test Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Optimize some code Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Fix test failure Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Modify yaml test description Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Remove the name parameter for setAnalyzer() Signed-off-by: Gao Binlong <gbinlong@amazon.com> --------- Signed-off-by: Gao Binlong <gbinlong@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Offline calculation of total shard per node and caching it for weight calculation inside LocalShardBalancer (opensearch-project#14675) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * [bug fix] validate lower bound for top n size (opensearch-project#14587) Signed-off-by: Chenyang Ji <cyji@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Create SystemIndexRegistry with helper method matchesSystemIndex (opensearch-project#14415) * Create new extension point in SystemIndexPlugin for a single plugin to get registered system indices Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add to CHANGELOG Signed-off-by: Craig Perkins <cwperx@amazon.com> * WIP on system indices from IndexNameExpressionResolver Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add test in IndexNameExpressionResolverTests Signed-off-by: Craig Perkins <cwperx@amazon.com> * Remove changes in SystemIndexPlugin Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add method in IndexNameExpressionResolver to get matching system indices Signed-off-by: Craig Perkins <cwperx@amazon.com> * Show how resolver can be chained to get system indices Signed-off-by: Craig Perkins <cwperx@amazon.com> * Fix forbiddenApis check Signed-off-by: Craig Perkins <cwperx@amazon.com> * Update CHANGELOG Signed-off-by: Craig Perkins <cwperx@amazon.com> * Make SystemIndices internal Signed-off-by: Craig Perkins <cwperx@amazon.com> * Remove unneeded changes Signed-off-by: Craig Perkins <cwperx@amazon.com> * Fix CI failures Signed-off-by: Craig Perkins <cwperx@amazon.com> * Fix precommit errors Signed-off-by: Craig Perkins <cwperx@amazon.com> * Use Regex instead of WildcardMatcher Signed-off-by: Craig Perkins <cwperx@amazon.com> * Address code review feedback Signed-off-by: Craig Perkins <cwperx@amazon.com> * Allow caller to pass index expressions Signed-off-by: Craig Perkins <cwperx@amazon.com> * Create SystemIndexRegistry Signed-off-by: Craig Perkins <cwperx@amazon.com> * Update CHANGELOG Signed-off-by: Craig Perkins <cwperx@amazon.com> * Remove singleton limitation Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add javadoc Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add @experimentalapi annotation Signed-off-by: Craig Perkins <cwperx@amazon.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Refactor Grok validate pattern to iterative approach (opensearch-project#14206) * grok validate patterns recusrion to iterative Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * Add max depth in resolving a pattern to avoid OOM Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * change path from deque to arraylist Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * rename queue to stack Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * Change max depth to 500 Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * typo originPatternName fix Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> * spotless Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> --------- Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Bump opentelemetry from 1.39.0 to 1.40.0 (opensearch-project#14674) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Bump jackson from 2.17.1 to 2.17.2 (opensearch-project#14687) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add release notes for release 1.3.18 (opensearch-project#14699) Signed-off-by: Zelin Hao <zelinhao@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Bump reactor from 3.5.19 to 3.5.20 (opensearch-project#14697) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add unit tests for read flow of RemoteClusterStateService and bug fix for transient settings (opensearch-project#14476) Signed-off-by: Shivansh Arora <hishiv@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Update version check for the bug fix of match_phrase_prefix_query not working on text field with multiple values and index_prefixes (opensearch-project#14703) Signed-off-by: Gao Binlong <gbinlong@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Remove unnecessary cast to int from test (opensearch-project#14696) Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * print reason why parent task was cancelled (opensearch-project#14604) Signed-off-by: kkewwei <kkewwei@163.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Use set of shard routing for shard in unassigned shard batch check. (opensearch-project#14533) Signed-off-by: Swetha Guptha <gupthasg@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add versioning for UploadedIndexMetadata (opensearch-project#14677) * Add versioning for UploadedIndexMetadata * Handle componentPrefix for backward compatibility Signed-off-by: Sooraj Sinha <soosinha@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Fix: update help output for _cat (opensearch-project#14722) * fixed help output for _cat Signed-off-by: ahmedsobeh <ahmed.sobeh@aiven.io> * updated changelog Signed-off-by: ahmedsobeh <ahmed.sobeh@aiven.io> * updated changelog Signed-off-by: ahmedsobeh <ahmed.sobeh@aiven.io> --------- Signed-off-by: ahmedsobeh <ahmed.sobeh@aiven.io> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Fix hdfs-fixture kerb-admin & hadoop-minicluster dependencies are not being updated / false positive reports on CVEs (opensearch-project#14729) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Update to Gradle 8.9 (opensearch-project#14574) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Fix hdfs-fixture hadoop-minicluster dependencies are not being updated / false positive reports on CVEs (opensearch-project#14732) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add `strict_allow_templates` dynamic mapping option (opensearch-project#14555) * The dynamic mapping parameter supports strict_allow_templates Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Modify change log Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Modify skip version in yml test file Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Refactor some code Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Keep the old methods Signed-off-by: Gao Binlong <gbinlong@amazon.com> * change public to private Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Optimize some code Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Do not override toString method for Dynamic Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Optimize some code and modify the changelog Signed-off-by: Gao Binlong <gbinlong@amazon.com> --------- Signed-off-by: Gao Binlong <gbinlong@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Bump net.minidev:json-smart from 2.5.0 to 2.5.1 in /plugins/repository-azure (opensearch-project#14748) * Bump net.minidev:json-smart in /plugins/repository-azure Bumps [net.minidev:json-smart](https://github.com/netplex/json-smart-v2) from 2.5.0 to 2.5.1. - [Release notes](https://github.com/netplex/json-smart-v2/releases) - [Commits](netplex/json-smart-v2@2.5.0...2.5.1) --- updated-dependencies: - dependency-name: net.minidev:json-smart dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * remove query insights plugin from core (opensearch-project#14743) Signed-off-by: Chenyang Ji <cyji@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add `strict_allow_templates` dynamic mapping option (opensearch-project#14555) (opensearch-project#14737) (opensearch-project#14742) * The dynamic mapping parameter supports strict_allow_templates * Modify change log * Modify skip version in yml test file * Refactor some code * Keep the old methods * change public to private * Optimize some code * Do not override toString method for Dynamic * Optimize some code and modify the changelog --------- (cherry picked from commit 6b8b3ef) Signed-off-by: Gao Binlong <gbinlong@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Fix create or update alias API doesn't throw exception for unsupported parameters (opensearch-project#14719) * Fix create or update alias API doesn't throw exception for unsupported parameters Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Update version check in yml test Signed-off-by: Gao Binlong <gbinlong@amazon.com> * modify change log Signed-off-by: Gao Binlong <gbinlong@amazon.com> --------- Signed-off-by: Gao Binlong <gbinlong@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Remove query categorization from core (opensearch-project#14759) * Remove query categorization from core Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Add changelog Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Trigger Build Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> --------- Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add changes to propagate queryGroupId across child requests and nodes (opensearch-project#14614) * add query group header propagator Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * apply spotless check Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add new propagator in ThreadContext Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * spotlessApply Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Bump com.microsoft.azure:msal4j from 1.15.1 to 1.16.0 in /plugins/repository-azure (opensearch-project#14610) * Bump com.microsoft.azure:msal4j in /plugins/repository-azure Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.15.1 to 1.16.0. - [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-java/releases) - [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt) - [Commits](AzureAD/microsoft-authentication-library-for-java@v1.15.1...v1.16.0) --- updated-dependencies: - dependency-name: com.microsoft.azure:msal4j dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * [Bugfix] Fix ICacheKeySerializerTests flakiness (opensearch-project#14564) * Fix testInvalidInput flakiness Signed-off-by: Peter Alfonsi <petealft@amazon.com> * Addressed andrross's comment Signed-off-by: Peter Alfonsi <petealft@amazon.com> * rerun security check Signed-off-by: Peter Alfonsi <petealft@amazon.com> --------- Signed-off-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Correct typo in method name (opensearch-project#14621) Signed-off-by: vatsal <vatsal.v.anand@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Refactoring FilterPath.parse by using an iterative approach instead of recursion. (opensearch-project#14200) * Refactor FilterPath parse function (opensearch-project#12067) Signed-off-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Implement unit tests for FilterPathTests (opensearch-project#12067) Signed-off-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Write warn log if Filter is empty; Add comments (opensearch-project#12067) Signed-off-by: Robin Friedmann <robinfriedmann.rf@gmail.com> * Add changelog Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Remove unnecessary log statement Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Remove unused logger Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Spotless apply Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Remove incorrect changelog Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> --------- Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> Co-authored-by: Robin Friedmann <robinfriedmann.rf@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Removing String format in RemoteStoreMigrationAllocationDecider to optimise performance(opensearch-project#14612) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata; Correct the check for deciding upload of HashesOfConsistentSettings (opensearch-project#14513) * Clear templates before Adding; Use NamedWriteableAwareStreamInput for RemoteCustomMetadata * Correct the check for deciding upload of hashes of consistent settings Signed-off-by: Sooraj Sinha <soosinha@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add changelog Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add PR link changelog Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Improve reroute performance by optimising List.removeAll in LocalShardsBalancer to filter remote search shard from relocation decision (opensearch-project#14613) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Fix assertion failure while deleting remote backed index (opensearch-project#14601) Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Allow system index warning in OpenSearchRestTestCase.refreshAllIndices (opensearch-project#14635) * Allow system index warning Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add to CHANGELOG Signed-off-by: Craig Perkins <cwperx@amazon.com> * Address code review comments Signed-off-by: Craig Perkins <cwperx@amazon.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Star tree codec changes (opensearch-project#14514) --------- Signed-off-by: Bharathwaj G <bharath78910@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Bump com.github.spullara.mustache.java:compiler from 0.9.13 to 0.9.14 in /modules/lang-mustache (opensearch-project#14672) * Bump com.github.spullara.mustache.java:compiler Bumps [com.github.spullara.mustache.java:compiler](https://github.com/spullara/mustache.java) from 0.9.13 to 0.9.14. - [Commits](spullara/mustache.java@mustache.java-0.9.13...mustache.java-0.9.14) --- updated-dependencies: - dependency-name: com.github.spullara.mustache.java:compiler dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Bump net.minidev:accessors-smart from 2.5.0 to 2.5.1 in /plugins/repository-azure (opensearch-project#14673) * Bump net.minidev:accessors-smart in /plugins/repository-azure Bumps [net.minidev:accessors-smart](https://github.com/netplex/json-smart-v2) from 2.5.0 to 2.5.1. - [Release notes](https://github.com/netplex/json-smart-v2/releases) - [Commits](netplex/json-smart-v2@2.5.0...2.5.1) --- updated-dependencies: - dependency-name: net.minidev:accessors-smart dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * move query group thread context propagator out of ThreadContext Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> --------- Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Peter Alfonsi <petealft@amazon.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> Signed-off-by: Sooraj Sinha <soosinha@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Craig Perkins <cwperx@amazon.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Peter Alfonsi <peter.alfonsi@gmail.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: Vatsal <36672090+imvtsl@users.noreply.github.com> Co-authored-by: Siddhant Deshmukh <deshsid@amazon.com> Co-authored-by: Robin Friedmann <robinfriedmann.rf@gmail.com> Co-authored-by: rishavz_sagar <rishavsagar4b1@gmail.com> Co-authored-by: Sooraj Sinha <81695996+soosinha@users.noreply.github.com> Co-authored-by: Sachin Kale <sachinpkale@gmail.com> Co-authored-by: Craig Perkins <cwperx@amazon.com> Co-authored-by: Bharathwaj G <bharath78910@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add consumers to remote store based index settings (opensearch-project#14764) Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add matchesPluginSystemIndexPattern to SystemIndexRegistry (opensearch-project#14750) * Add matchesPluginSystemIndexPattern to SystemIndexRegistry Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add to CHANGELOG Signed-off-by: Craig Perkins <cwperx@amazon.com> * Use single data structure to keep track of system indices Signed-off-by: Craig Perkins <cwperx@amazon.com> * Address code review comments Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add test for getAllDescriptors Signed-off-by: Craig Perkins <cwperx@amazon.com> * Update server/src/main/java/org/opensearch/indices/SystemIndexRegistry.java Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Craig Perkins <craig5008@gmail.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com> Signed-off-by: Craig Perkins <craig5008@gmail.com> Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * SPI for loading ABC templates (opensearch-project#14659) * SPI for loading ABC templates Signed-off-by: mgodwan <mgodwan@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Fix bulk upsert ignores the default_pipeline and final_pipeline when the auto-created index matches the index template (opensearch-project#12891) * Fix bulk upsert ignores the default_pipeline and final_pipeline when auto-created index matches with the index template Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Modify changelog & comment Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Use new approach Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Fix test failure Signed-off-by: Gao Binlong <gbinlong@amazon.com> --------- Signed-off-by: Gao Binlong <gbinlong@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Fix flaky test due to node being used across all tests (opensearch-project#14787) Signed-off-by: Mohit Godwani <mgodwan@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Star Tree Implementation [OnHeap] (opensearch-project#14512) --------- Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add Gao Binlong as maintainer (opensearch-project#14796) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Clear ehcache disk cache files during initialization (opensearch-project#14738) * Clear ehcache disk cache files during initialization Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Adding UT to fix line coverage Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comment Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Adding more Uts for better line coverage Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Throwing exception in case we fail to clear cache files during startup Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Adding more UTs Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Adding a UT for more coverage Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing gradle build Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Update ehcache disk cache close() logic Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Refactor remote-routing-table service inline with remote state interfaces (opensearch-project#14668) --------- Signed-off-by: Arpit Bandejiya <abandeji@amazon.com> Signed-off-by: Arpit-Bandejiya <abandeji@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Set version to 2.15 for determining metadata during migration to remote store Signed-off-by: Sandeep Kumawat <skumwt@amazon.com> Co-authored-by: Sandeep Kumawat <skumwt@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Fix bulk upsert ignores the default_pipeline and final_pipeline when the auto-created index matches the index template (opensearch-project#14793) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Fix create or update alias API doesn't throw exception for unsupported parameters (opensearch-project#14769) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Change RCSS info logs to debug (opensearch-project#14814) Signed-off-by: Shivansh Arora <hishiv@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * [Bugfix] Fix NPE in ReplicaShardAllocator (opensearch-project#13993) (opensearch-project#14385) * [Bugfix] Fix NPE in ReplicaShardAllocator (opensearch-project#13993) Signed-off-by: Daniil Roman <daniilroman.cv@gmail.com> * Add fix info to CHANGELOG.md Signed-off-by: Daniil Roman <danroman17397@gmail.com> --------- Signed-off-by: Daniil Roman <daniilroman.cv@gmail.com> Signed-off-by: Daniil Roman <danroman17397@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Run performance benchmark on pull requests (opensearch-project#14760) * add performance benchmark workflow for pull requests Signed-off-by: Rishabh Singh <sngri@amazon.com> * Update PERFORMANCE_BENCHMARKS.md Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Rishabh Singh <rishabhksingh@gmail.com> * Update PERFORMANCE_BENCHMARKS.md Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Rishabh Singh <rishabhksingh@gmail.com> * Update .github/workflows/benchmark-pull-request.yml Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Rishabh Singh <rishabhksingh@gmail.com> * Update .github/workflows/benchmark-pull-request.yml Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Rishabh Singh <rishabhksingh@gmail.com> * Update .github/workflows/benchmark-pull-request.yml Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Rishabh Singh <rishabhksingh@gmail.com> * Update .github/workflows/benchmark-pull-request.yml Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Rishabh Singh <rishabhksingh@gmail.com> --------- Signed-off-by: Rishabh Singh <sngri@amazon.com> Signed-off-by: Rishabh Singh <rishabhksingh@gmail.com> Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix constant_keyword field type (opensearch-project#14807) Signed-off-by: kkewwei <kkewwei@163.com> test Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * [Remote Store Migration] Reconcile remote store based index settings during STRICT mode switch (opensearch-project#14792) Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add prefix mode verification setting for repository verification (opensearch-project#14790) * Add prefix mode verification setting for repository verification Signed-off-by: Ashish Singh <ssashish@amazon.com> * Add UTs and randomise prefix mode repository verification Signed-off-by: Ashish Singh <ssashish@amazon.com> * Incorporate PR review feedback Signed-off-by: Ashish Singh <ssashish@amazon.com> --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add length check on comment body for benchmark workflow (opensearch-project#14834) Signed-off-by: Rishabh Singh <sngri@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add restore-from-snapshot test procedure for snapshot run benchmark config (opensearch-project#14842) Signed-off-by: Rishabh Singh <sngri@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Fix env variable name typo (opensearch-project#14843) Signed-off-by: Rishabh Singh <sngri@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Use circuit breaker in InternalHistogram when adding empty buckets (opensearch-project#14754) * introduce circuit breaker in InternalHistogram Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * use circuit breaker from reduce context Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * add test Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * revert use_real_memory change in OpenSearchNode Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * add change log Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * [Remote State] Create interface RemoteEntitiesManager (opensearch-project#14671) * Create interface RemoteEntitiesManager Signed-off-by: Shivansh Arora <hishiv@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Optimise TransportNodesAction to not send DiscoveryNodes for NodeStat… (opensearch-project#14749) * Optimize TransportNodesAction to not send DiscoveryNodes for NodeStats, NodesInfo and ClusterStats call Signed-off-by: Pranshu Shukla <pranshushukla06@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Enabling term version check on local state for all ClusterManager Read Transport Actions (opensearch-project#14273) * enabling term version check on local state for all admin read actions Signed-off-by: Rajiv Kumar Vaidyanathan <rajivkv@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Reduce logging in DEBUG for MasterService:run (opensearch-project#14795) * Reduce logging in DEBUG for MasteService:run by introducing short and long summary in Taskbatcher Signed-off-by: Sumit Bansal <sumitsb@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add SplitResponseProcessor to Search Pipelines (opensearch-project#14800) * Add SplitResponseProcessor for search pipelines Signed-off-by: Daniel Widdis <widdis@gmail.com> * Register the split processor factory Signed-off-by: Daniel Widdis <widdis@gmail.com> * Address code review comments Signed-off-by: Daniel Widdis <widdis@gmail.com> * Avoid list copy by casting array Signed-off-by: Daniel Widdis <widdis@gmail.com> --------- Signed-off-by: Daniel Widdis <widdis@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add integration tests for RemoteRoutingTable Service. (opensearch-project#14631) Signed-off-by: Shailendra Singh <singhlhs@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add SortResponseProcessor to Search Pipelines (opensearch-project#14785) * Add SortResponseProcessor for search pipelines Signed-off-by: Daniel Widdis <widdis@gmail.com> * Add stupid and unnecessary javadocs to satisfy overly strict CI Signed-off-by: Daniel Widdis <widdis@gmail.com> * Split casting and sorting methods for readability Signed-off-by: Daniel Widdis <widdis@gmail.com> * Register the sort processor factory Signed-off-by: Daniel Widdis <widdis@gmail.com> * Address code review comments Signed-off-by: Daniel Widdis <widdis@gmail.com> * Cast individual list elements to avoid creating two lists Signed-off-by: Daniel Widdis <widdis@gmail.com> * Add yamlRestTests Signed-off-by: Daniel Widdis <widdis@gmail.com> * Clarify why there's unusual sorting Signed-off-by: Daniel Widdis <widdis@gmail.com> * Use instanceof instead of isAssignableFrom Signed-off-by: Daniel Widdis <widdis@gmail.com> --------- Signed-off-by: Daniel Widdis <widdis@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Fix allowUnmappedFields, mapUnmappedFieldAsString settings to be applied when parsing query string query (opensearch-project#13957) * Modify to invoke QueryShardContext.fieldMapper() method to apply allowUnmappedFields and mapUnmappedFieldAsString settings Signed-off-by: imyp92 <pyw5420@gmail.com> * Add test cases to verify returning 400 responses if unmapped fields are included for some types of query Signed-off-by: imyp92 <pyw5420@gmail.com> * Add changelog Signed-off-by: imyp92 <pyw5420@gmail.com> --------- Signed-off-by: imyp92 <pyw5420@gmail.com> Signed-off-by: gaobinlong <gbinlong@amazon.com> Co-authored-by: gaobinlong <gbinlong@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Bump com.microsoft.azure:msal4j from 1.16.0 to 1.16.1 in /plugins/repository-azure (opensearch-project#14857) * Bump com.microsoft.azure:msal4j in /plugins/repository-azure Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.16.0 to 1.16.1. - [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-java/releases) - [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt) - [Commits](AzureAD/microsoft-authentication-library-for-java@v1.16.0...v1.16.1) --- updated-dependencies: - dependency-name: com.microsoft.azure:msal4j dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Bump com.gradle.develocity from 3.17.5 to 3.17.6 (opensearch-project#14856) * Bump com.gradle.develocity from 3.17.5 to 3.17.6 Bumps com.gradle.develocity from 3.17.5 to 3.17.6. --- updated-dependencies: - dependency-name: com.gradle.develocity dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Bump org.jline:jline in /test/fixtures/hdfs-fixture (opensearch-project#14859) Bumps [org.jline:jline](https://github.com/jline/jline3) from 3.26.2 to 3.26.3. - [Release notes](https://github.com/jline/jline3/releases) - [Changelog](https://github.com/jline/jline3/blob/master/changelog.md) - [Commits](jline/jline3@jline-parent-3.26.2...jline-parent-3.26.3) --- updated-dependencies: - dependency-name: org.jline:jline dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Use Lucene provided Persian stem (opensearch-project#14847) Lucene provided Persian stem apparently isn't hooked yet and this change is doing that based on what is done for Arabic stem support. Signed-off-by: Ebrahim Byagowi <ebrahim@gnu.org> Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Bump actions/checkout from 2 to 4 (opensearch-project#14858) * Bump actions/checkout from 2 to 4 Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 4. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v2...v4) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Deprecate batch_size parameter on bulk API (opensearch-project#14725) By default the full _bulk payload will be passed to ingest processors as a batch, with any sub batching logic to be implemented by each processor if necessary. Signed-off-by: Liyun Xiu <xiliyun@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add perms for remote snapshot cache eviction on scripted query (opensearch-project#14411) Signed-off-by: Finn Carroll <carrofin@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add transport interceptor to populate queryGroupId in task headers Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Add rest, transport layer changes for Hot to warm tiering - dedicated setup (opensearch-project#13980) Signed-off-by: Neetika Singhal <neetiks@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Create listener to refresh search thread resource usage (opensearch-project#14832) * [bug fix] fix incorrect coordinator node search resource usages Signed-off-by: Chenyang Ji <cyji@amazon.com> * fix bug on serialization when passing task resource usage to coordinator Signed-off-by: Chenyang Ji <cyji@amazon.com> * add more unit tests Signed-off-by: Chenyang Ji <cyji@amazon.com> * remove query insights plugin related code Signed-off-by: Chenyang Ji <cyji@amazon.com> * create per request listener to refresh task resource usage Signed-off-by: Chenyang Ji <cyji@amazon.com> * Make new listener API public Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Add changelog Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Remove wrong files added Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Address review comments Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Build fix Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Make singleton Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Address review comments Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Make sure listener runs before plugin listeners Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Spotless Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Minor fix Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> --------- Signed-off-by: Chenyang Ji <cyji@amazon.com> Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> Signed-off-by: Jay Deng <jayd0104@gmail.com> Co-authored-by: Chenyang Ji <cyji@amazon.com> Co-authored-by: Jay Deng <jayd0104@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Caching avg total bytes and avg free bytes inside ClusterInfo (opensearch-project#14851) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Use default value when index.number_of_replicas is null (opensearch-project#14812) * Use default value when index.number_of_replicas is null Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Add integration test Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Add changelog Signed-off-by: Liyun Xiu <xiliyun@amazon.com> --------- Signed-off-by: Liyun Xiu <xiliyun@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * [Remote Routing Table] Implement write and read flow for shard diff file. (opensearch-project#14684) * Implement write and read flow to upload/download shard diff file. Signed-off-by: Shailendra Singh <singhlhs@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Optimized ClusterStatsIndices to precomute shard stats (opensearch-project#14426) * Optimize Cluster Stats Indices to precomute node level stats Signed-off-by: Pranshu Shukla <pranshushukla06@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Fix constraint bug which allows more primary shards than average primary shards per index (opensearch-project#14908) Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Optmising AwarenessAllocationDecider for hashmap.get call (opensearch-project#14761) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * update comment Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * Fix IngestServiceTests.testBulkRequestExecutionWithFailures (opensearch-project#14918) The test would previously fail if the randomness led to only a single indexing request being included in the bulk payload. This change guarantees multiple indexing requests in order to ensure the batch logic kicks in. Also replace some unneeded mocks with real classes. Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add queryGroupTask Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * remove unnecessary imports Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add QueryGroupTask tests Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * rename WLM transport request handler Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add CHANGELOG entry Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix ut Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix UT to remove the verify for final method Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * apply spotless Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> --------- Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com> Signed-off-by: Gao Binlong <gbinlong@amazon.com> Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com> Signed-off-by: Chenyang Ji <cyji@amazon.com> Signed-off-by: Craig Perkins <cwperx@amazon.com> Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Zelin Hao <zelinhao@amazon.com> Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io> Signed-off-by: kkewwei <kkewwei@163.com> Signed-off-by: Swetha Guptha <gupthasg@amazon.com> Signed-off-by: Sooraj Sinha <soosinha@amazon.com> Signed-off-by: ahmedsobeh <ahmed.sobeh@aiven.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> Signed-off-by: Peter Alfonsi <petealft@amazon.com> Signed-off-by: vatsal <vatsal.v.anand@gmail.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Craig Perkins <craig5008@gmail.com> Signed-off-by: mgodwan <mgodwan@amazon.com> Signed-off-by: Mohit Godwani <mgodwan@amazon.com> Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sandeep Kumawat <skumwt@amazon.com> Signed-off-by: Daniil Roman <daniilroman.cv@gmail.com> Signed-off-by: Daniil Roman <danroman17397@gmail.com> Signed-off-by: Rishabh Singh <sngri@amazon.com> Signed-off-by: Rishabh Singh <rishabhksingh@gmail.com> Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Pranshu Shukla <pranshushukla06@gmail.com> Signed-off-by: Rajiv Kumar Vaidyanathan <rajivkv@amazon.com> Signed-off-by: Sumit Bansal <sumitsb@amazon.com> Signed-off-by: Daniel Widdis <widdis@gmail.com> Signed-off-by: Shailendra Singh <singhlhs@amazon.com> Signed-off-by: imyp92 <pyw5420@gmail.com> Signed-off-by: gaobinlong <gbinlong@amazon.com> Signed-off-by: Ebrahim Byagowi <ebrahim@gnu.org> Signed-off-by: Liyun Xiu <xiliyun@amazon.com> Signed-off-by: Finn Carroll <carrofin@amazon.com> Signed-off-by: Neetika Singhal <neetiks@amazon.com> Signed-off-by: Jay Deng <jayd0104@gmail.com> Signed-off-by: Gaurav Bafna <gbbafna@amazon.com> Signed-off-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Shivansh Arora <hishiv@amazon.com> Co-authored-by: Arpit-Bandejiya <abandeji@amazon.com> Co-authored-by: gaobinlong <gbinlong@amazon.com> Co-authored-by: rishavz_sagar <rishavsagar4b1@gmail.com> Co-authored-by: Chenyang Ji <cyji@amazon.com> Co-authored-by: Craig Perkins <cwperx@amazon.com> Co-authored-by: Sandesh Kumar <sandeshkr419@gmail.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Zelin Hao <zelinhao@amazon.com> Co-authored-by: Lukáš Vlček <lukas.vlcek@aiven.io> Co-authored-by: kkewwei <kkewwei@163.com> Co-authored-by: SwethaGuptha <156877431+SwethaGuptha@users.noreply.github.com> Co-authored-by: Sooraj Sinha <81695996+soosinha@users.noreply.github.com> Co-authored-by: Ahmed Sobeh <ahmed.sobeh@aiven.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Siddhant Deshmukh <deshsid@amazon.com> Co-authored-by: Peter Alfonsi <peter.alfonsi@gmail.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: Vatsal <36672090+imvtsl@users.noreply.github.com> Co-authored-by: Robin Friedmann <robinfriedmann.rf@gmail.com> Co-authored-by: Sachin Kale <sachinpkale@gmail.com> Co-authored-by: Bharathwaj G <bharath78910@gmail.com> Co-authored-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Co-authored-by: Craig Perkins <craig5008@gmail.com> Co-authored-by: Andriy Redko <drreta@gmail.com> Co-authored-by: Mohit Godwani <81609427+mgodwan@users.noreply.github.com> Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com> Co-authored-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Sandeep Kumawat <2025sandeepkumawat@gmail.com> Co-authored-by: Sandeep Kumawat <skumwt@amazon.com> Co-authored-by: Daniil Roman <danroman17397@gmail.com> Co-authored-by: Rishabh Singh <rishabhksingh@gmail.com> Co-authored-by: kkewwei <kewei.11@bytedance.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> Co-authored-by: Ashish Singh <ssashish@amazon.com> Co-authored-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Pranshu Shukla <55992439+Pranshu-S@users.noreply.github.com> Co-authored-by: rajiv-kv <157019998+rajiv-kv@users.noreply.github.com> Co-authored-by: Sumit Bansal <sumit.asr@gmail.com> Co-authored-by: Daniel Widdis <widdis@gmail.com> Co-authored-by: shailendra0811 <167273922+shailendra0811@users.noreply.github.com> Co-authored-by: Park, Yeongwu <pyw5420@gmail.com> Co-authored-by: ebraminio <ebraminio@gmail.com> Co-authored-by: Liyun Xiu <xiliyun@amazon.com> Co-authored-by: Finn <carrofin@amazon.com> Co-authored-by: Neetika Singhal <neetiks@amazon.com> Co-authored-by: Jay Deng <jayd0104@gmail.com> Co-authored-by: Gaurav Bafna <85113518+gbbafna@users.noreply.github.com> Co-authored-by: Andrew Ross <andrross@amazon.com>
1 parent e67b9e1 commit c0e1d49

10 files changed

+320
-5
lines changed

‎CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
66
## [Unreleased 2.x]
77
### Added
88
- Fix for hasInitiatedFetching to fix allocation explain and manual reroute APIs (([#14972](https://github.com/opensearch-project/OpenSearch/pull/14972))
9+
- [Workload Management] Add queryGroupId to Task ([14708](https://github.com/opensearch-project/OpenSearch/pull/14708))
910
- Add basic aggregation support for derived fields ([#14618](https://github.com/opensearch-project/OpenSearch/pull/14618))
1011

1112
### Dependencies

‎server/src/main/java/org/opensearch/action/search/SearchShardTask.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,8 @@
3737
import org.opensearch.core.tasks.TaskId;
3838
import org.opensearch.search.fetch.ShardFetchSearchRequest;
3939
import org.opensearch.search.internal.ShardSearchRequest;
40-
import org.opensearch.tasks.CancellableTask;
4140
import org.opensearch.tasks.SearchBackpressureTask;
41+
import org.opensearch.wlm.QueryGroupTask;
4242

4343
import java.util.Map;
4444
import java.util.function.Supplier;
@@ -50,7 +50,7 @@
5050
* @opensearch.api
5151
*/
5252
@PublicApi(since = "1.0.0")
53-
public class SearchShardTask extends CancellableTask implements SearchBackpressureTask {
53+
public class SearchShardTask extends QueryGroupTask implements SearchBackpressureTask {
5454
// generating metadata in a lazy way since source can be quite big
5555
private final MemoizedSupplier<String> metadataSupplier;
5656

‎server/src/main/java/org/opensearch/action/search/SearchTask.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,8 @@
3535
import org.opensearch.common.annotation.PublicApi;
3636
import org.opensearch.common.unit.TimeValue;
3737
import org.opensearch.core.tasks.TaskId;
38-
import org.opensearch.tasks.CancellableTask;
3938
import org.opensearch.tasks.SearchBackpressureTask;
39+
import org.opensearch.wlm.QueryGroupTask;
4040

4141
import java.util.Map;
4242
import java.util.function.Supplier;
@@ -49,7 +49,7 @@
4949
* @opensearch.api
5050
*/
5151
@PublicApi(since = "1.0.0")
52-
public class SearchTask extends CancellableTask implements SearchBackpressureTask {
52+
public class SearchTask extends QueryGroupTask implements SearchBackpressureTask {
5353
// generating description in a lazy way since source can be quite big
5454
private final Supplier<String> descriptionSupplier;
5555
private SearchProgressListener progressListener = SearchProgressListener.NOOP;

‎server/src/main/java/org/opensearch/action/search/TransportSearchAction.java

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,7 @@
101101
import org.opensearch.transport.RemoteTransportException;
102102
import org.opensearch.transport.Transport;
103103
import org.opensearch.transport.TransportService;
104+
import org.opensearch.wlm.QueryGroupTask;
104105

105106
import java.util.ArrayList;
106107
import java.util.Arrays;
@@ -442,6 +443,12 @@ private void executeRequest(
442443
);
443444
searchRequestContext.getSearchRequestOperationsListener().onRequestStart(searchRequestContext);
444445

446+
// At this point either the QUERY_GROUP_ID header will be present in ThreadContext either via ActionFilter
447+
// or HTTP header (HTTP header will be deprecated once ActionFilter is implemented)
448+
if (task instanceof QueryGroupTask) {
449+
((QueryGroupTask) task).setQueryGroupId(threadPool.getThreadContext());
450+
}
451+
445452
PipelinedRequest searchRequest;
446453
ActionListener<SearchResponse> listener;
447454
try {

‎server/src/main/java/org/opensearch/node/Node.java

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -263,6 +263,7 @@
263263
import org.opensearch.transport.TransportService;
264264
import org.opensearch.usage.UsageService;
265265
import org.opensearch.watcher.ResourceWatcherService;
266+
import org.opensearch.wlm.WorkloadManagementTransportInterceptor;
266267

267268
import javax.net.ssl.SNIHostName;
268269

@@ -1047,14 +1048,21 @@ protected Node(
10471048
admissionControlService
10481049
);
10491050

1051+
WorkloadManagementTransportInterceptor workloadManagementTransportInterceptor = new WorkloadManagementTransportInterceptor(
1052+
threadPool
1053+
);
1054+
10501055
final Collection<SecureSettingsFactory> secureSettingsFactories = pluginsService.filterPlugins(Plugin.class)
10511056
.stream()
10521057
.map(p -> p.getSecureSettingFactory(settings))
10531058
.filter(Optional::isPresent)
10541059
.map(Optional::get)
10551060
.collect(Collectors.toList());
10561061

1057-
List<TransportInterceptor> transportInterceptors = List.of(admissionControlTransportInterceptor);
1062+
List<TransportInterceptor> transportInterceptors = List.of(
1063+
admissionControlTransportInterceptor,
1064+
workloadManagementTransportInterceptor
1065+
);
10581066
final NetworkModule networkModule = new NetworkModule(
10591067
settings,
10601068
pluginsService.filterPlugins(NetworkPlugin.class),

‎server/src/main/java/org/opensearch/wlm/QueryGroupTask.java

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.wlm;
10+
11+
import org.apache.logging.log4j.LogManager;
12+
import org.apache.logging.log4j.Logger;
13+
import org.opensearch.common.unit.TimeValue;
14+
import org.opensearch.common.util.concurrent.ThreadContext;
15+
import org.opensearch.core.tasks.TaskId;
16+
import org.opensearch.tasks.CancellableTask;
17+
18+
import java.util.Map;
19+
import java.util.Optional;
20+
import java.util.function.Supplier;
21+
22+
import static org.opensearch.search.SearchService.NO_TIMEOUT;
23+
24+
/**
25+
* Base class to define QueryGroup tasks
26+
*/
27+
public class QueryGroupTask extends CancellableTask {
28+
29+
private static final Logger logger = LogManager.getLogger(QueryGroupTask.class);
30+
public static final String QUERY_GROUP_ID_HEADER = "queryGroupId";
31+
public static final Supplier<String> DEFAULT_QUERY_GROUP_ID_SUPPLIER = () -> "DEFAULT_QUERY_GROUP";
32+
private String queryGroupId;
33+
34+
public QueryGroupTask(long id, String type, String action, String description, TaskId parentTaskId, Map<String, String> headers) {
35+
this(id, type, action, description, parentTaskId, headers, NO_TIMEOUT);
36+
}
37+
38+
public QueryGroupTask(
39+
long id,
40+
String type,
41+
String action,
42+
String description,
43+
TaskId parentTaskId,
44+
Map<String, String> headers,
45+
TimeValue cancelAfterTimeInterval
46+
) {
47+
super(id, type, action, description, parentTaskId, headers, cancelAfterTimeInterval);
48+
}
49+
50+
/**
51+
* This method should always be called after calling setQueryGroupId at least once on this object
52+
* @return task queryGroupId
53+
*/
54+
public final String getQueryGroupId() {
55+
if (queryGroupId == null) {
56+
logger.warn("QueryGroup _id can't be null, It should be set before accessing it. This is abnormal behaviour ");
57+
}
58+
return queryGroupId;
59+
}
60+
61+
/**
62+
* sets the queryGroupId from threadContext into the task itself,
63+
* This method was defined since the queryGroupId can only be evaluated after task creation
64+
* @param threadContext current threadContext
65+
*/
66+
public final void setQueryGroupId(final ThreadContext threadContext) {
67+
this.queryGroupId = Optional.ofNullable(threadContext)
68+
.map(threadContext1 -> threadContext1.getHeader(QUERY_GROUP_ID_HEADER))
69+
.orElse(DEFAULT_QUERY_GROUP_ID_SUPPLIER.get());
70+
}
71+
72+
@Override
73+
public boolean shouldCancelChildrenOnCancellation() {
74+
return false;
75+
}
76+
}

‎server/src/main/java/org/opensearch/wlm/WorkloadManagementTransportInterceptor.java

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.wlm;
10+
11+
import org.opensearch.tasks.Task;
12+
import org.opensearch.threadpool.ThreadPool;
13+
import org.opensearch.transport.TransportChannel;
14+
import org.opensearch.transport.TransportInterceptor;
15+
import org.opensearch.transport.TransportRequest;
16+
import org.opensearch.transport.TransportRequestHandler;
17+
18+
/**
19+
* This class is used to intercept search traffic requests and populate the queryGroupId header in task headers
20+
*/
21+
public class WorkloadManagementTransportInterceptor implements TransportInterceptor {
22+
private final ThreadPool threadPool;
23+
24+
public WorkloadManagementTransportInterceptor(ThreadPool threadPool) {
25+
this.threadPool = threadPool;
26+
}
27+
28+
@Override
29+
public <T extends TransportRequest> TransportRequestHandler<T> interceptHandler(
30+
String action,
31+
String executor,
32+
boolean forceExecution,
33+
TransportRequestHandler<T> actualHandler
34+
) {
35+
return new RequestHandler<T>(threadPool, actualHandler);
36+
}
37+
38+
/**
39+
* This class is mainly used to populate the queryGroupId header
40+
* @param <T> T is Search related request
41+
*/
42+
public static class RequestHandler<T extends TransportRequest> implements TransportRequestHandler<T> {
43+
44+
private final ThreadPool threadPool;
45+
TransportRequestHandler<T> actualHandler;
46+
47+
public RequestHandler(ThreadPool threadPool, TransportRequestHandler<T> actualHandler) {
48+
this.threadPool = threadPool;
49+
this.actualHandler = actualHandler;
50+
}
51+
52+
@Override
53+
public void messageReceived(T request, TransportChannel channel, Task task) throws Exception {
54+
if (isSearchWorkloadRequest(task)) {
55+
((QueryGroupTask) task).setQueryGroupId(threadPool.getThreadContext());
56+
}
57+
actualHandler.messageReceived(request, channel, task);
58+
}
59+
60+
boolean isSearchWorkloadRequest(Task task) {
61+
return task instanceof QueryGroupTask;
62+
}
63+
}
64+
}

‎server/src/test/java/org/opensearch/wlm/QueryGroupTaskTests.java

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.wlm;
10+
11+
import org.opensearch.test.OpenSearchTestCase;
12+
import org.opensearch.threadpool.TestThreadPool;
13+
import org.opensearch.threadpool.ThreadPool;
14+
15+
import java.util.Collections;
16+
17+
import static org.opensearch.wlm.QueryGroupTask.DEFAULT_QUERY_GROUP_ID_SUPPLIER;
18+
import static org.opensearch.wlm.QueryGroupTask.QUERY_GROUP_ID_HEADER;
19+
20+
public class QueryGroupTaskTests extends OpenSearchTestCase {
21+
private ThreadPool threadPool;
22+
private QueryGroupTask sut;
23+
24+
public void setUp() throws Exception {
25+
super.setUp();
26+
threadPool = new TestThreadPool(getTestName());
27+
sut = new QueryGroupTask(123, "transport", "Search", "test task", null, Collections.emptyMap());
28+
}
29+
30+
public void tearDown() throws Exception {
31+
super.tearDown();
32+
threadPool.shutdown();
33+
}
34+
35+
public void testSuccessfulSetQueryGroupId() {
36+
sut.setQueryGroupId(threadPool.getThreadContext());
37+
assertEquals(DEFAULT_QUERY_GROUP_ID_SUPPLIER.get(), sut.getQueryGroupId());
38+
39+
threadPool.getThreadContext().putHeader(QUERY_GROUP_ID_HEADER, "akfanglkaglknag2332");
40+
41+
sut.setQueryGroupId(threadPool.getThreadContext());
42+
assertEquals("akfanglkaglknag2332", sut.getQueryGroupId());
43+
}
44+
}

‎server/src/test/java/org/opensearch/wlm/WorkloadManagementTransportInterceptorTests.java

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.wlm;
10+
11+
import org.opensearch.test.OpenSearchTestCase;
12+
import org.opensearch.threadpool.TestThreadPool;
13+
import org.opensearch.threadpool.ThreadPool;
14+
import org.opensearch.transport.TransportRequest;
15+
import org.opensearch.transport.TransportRequestHandler;
16+
import org.opensearch.wlm.WorkloadManagementTransportInterceptor.RequestHandler;
17+
18+
import static org.opensearch.threadpool.ThreadPool.Names.SAME;
19+
20+
public class WorkloadManagementTransportInterceptorTests extends OpenSearchTestCase {
21+
22+
private ThreadPool threadPool;
23+
private WorkloadManagementTransportInterceptor sut;
24+
25+
public void setUp() throws Exception {
26+
super.setUp();
27+
threadPool = new TestThreadPool(getTestName());
28+
sut = new WorkloadManagementTransportInterceptor(threadPool);
29+
}
30+
31+
public void tearDown() throws Exception {
32+
super.tearDown();
33+
threadPool.shutdown();
34+
}
35+
36+
public void testInterceptHandler() {
37+
TransportRequestHandler<TransportRequest> requestHandler = sut.interceptHandler("Search", SAME, false, null);
38+
assertTrue(requestHandler instanceof RequestHandler);
39+
}
40+
}

0 commit comments

Comments
 (0)