Skip to content

Commit 1c5f519

Browse files
author
meetvm
committed
Fix Issue 16890: Hide Stracktrace while translog transfer
Signed-off-by: meetvm <meetvm@amazon.com> Bump com.nimbusds:oauth2-oidc-sdk from 11.19.1 to 11.20.1 in /plugins/repository-azure (opensearch-project#16895) * Bump com.nimbusds:oauth2-oidc-sdk in /plugins/repository-azure Bumps [com.nimbusds:oauth2-oidc-sdk](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions) from 11.19.1 to 11.20.1. - [Changelog](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions/src/master/CHANGELOG.txt) - [Commits](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions/branches/compare/11.20.1..11.19.1) --- updated-dependencies: - dependency-name: com.nimbusds:oauth2-oidc-sdk dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Bump com.netflix.nebula.ospackage-base from 11.10.0 to 11.10.1 in /distribution/packages (opensearch-project#16896) * Bump com.netflix.nebula.ospackage-base in /distribution/packages Bumps com.netflix.nebula.ospackage-base from 11.10.0 to 11.10.1. --- updated-dependencies: - dependency-name: com.netflix.nebula.ospackage-base dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Bump ch.qos.logback:logback-classic from 1.5.12 to 1.5.15 in /test/fixtures/hdfs-fixture (opensearch-project#16898) * Bump ch.qos.logback:logback-classic in /test/fixtures/hdfs-fixture Bumps [ch.qos.logback:logback-classic](https://github.com/qos-ch/logback) from 1.5.12 to 1.5.15. - [Commits](qos-ch/logback@v_1.5.12...v_1.5.15) --- updated-dependencies: - dependency-name: ch.qos.logback:logback-classic dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Bump lycheeverse/lychee-action from 2.1.0 to 2.2.0 (opensearch-project#16897) * Bump lycheeverse/lychee-action from 2.1.0 to 2.2.0 Bumps [lycheeverse/lychee-action](https://github.com/lycheeverse/lychee-action) from 2.1.0 to 2.2.0. - [Release notes](https://github.com/lycheeverse/lychee-action/releases) - [Commits](lycheeverse/lychee-action@v2.1.0...v2.2.0) --- updated-dependencies: - dependency-name: lycheeverse/lychee-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Create sub directories for ThirdPartyAudit dependency metadata (opensearch-project#16844) * Extract jars to sub dirs during thirdPartyAudit task. Signed-off-by: Finn Carroll <carrofin@amazon.com> * Change regex to split on '-'/'.'. Ignore version. Signed-off-by: Finn Carroll <carrofin@amazon.com> * Split on .jar for sub folder prefix. Signed-off-by: Finn Carroll <carrofin@amazon.com> --------- Signed-off-by: Finn Carroll <carrofin@amazon.com> Retrieve value from DocValues in a flat_object filed (opensearch-project#16802) Bump com.microsoft.azure:msal4j from 1.17.2 to 1.18.0 in /plugins/repository-azure (opensearch-project#16918) * Bump com.microsoft.azure:msal4j in /plugins/repository-azure Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.17.2 to 1.18.0. - [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-java/releases) - [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt) - [Commits](AzureAD/microsoft-authentication-library-for-java@v1.17.2...v1.18.0) --- updated-dependencies: - dependency-name: com.microsoft.azure:msal4j dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Bump org.apache.commons:commons-text from 1.12.0 to 1.13.0 in /test/fixtures/hdfs-fixture (opensearch-project#16919) * Bump org.apache.commons:commons-text in /test/fixtures/hdfs-fixture Bumps org.apache.commons:commons-text from 1.12.0 to 1.13.0. --- updated-dependencies: - dependency-name: org.apache.commons:commons-text dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Add gRPC server as transport-grpc plugin (opensearch-project#16534) Introduce auxiliary transport to NetworkPlugin and add gRPC plugin. Auxiliary transports are optional lifecycle components provided by network plugins which run in parallel to the http server/native transport. They are distinct from the existing NetworkPlugin interfaces of 'getTransports' and 'getHttpTransports' as auxiliary transports are optional. Each AuxTransport implements it's own 'aux.transport.type' and 'aux.transport.<type>.ports' setting. Since Security.java initializes previous to Node.java during bootstrap socket binding permissions are granted based on 'aux.transport.<type>.ports' for each enabled 'aux.transport.type', falling back to a default if no ports are specified. Signed-off-by: Finn Carroll <carrofin@amazon.com> Update script supports java.lang.String.sha1() and java.lang.String.sha256() methods (opensearch-project#16923) * Update script supports java.lang.String.sha1() and java.lang.String.sha256() methods Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Modify change log Signed-off-by: Gao Binlong <gbinlong@amazon.com> --------- Signed-off-by: Gao Binlong <gbinlong@amazon.com> Workflow benchmark-pull-request.yml fix (opensearch-project#16925) Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> Add benchmark confirm for lucene-10 big5 index snapshot (opensearch-project#16940) Signed-off-by: Rishabh Singh <sngri@amazon.com> Remove duplicate DCO check (opensearch-project#16942) Signed-off-by: Andriy Redko <drreta@gmail.com> Allow extended plugins to be optional (opensearch-project#16909) * Make extended plugins optional Signed-off-by: Craig Perkins <cwperx@amazon.com> * Make extended plugins optional Signed-off-by: Craig Perkins <cwperx@amazon.com> * Load extensions for classpath plugins Signed-off-by: Craig Perkins <cwperx@amazon.com> * Ensure only single instance for each classpath extension Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add test for classpath plugin extended plugin loading Signed-off-by: Craig Perkins <cwperx@amazon.com> * Modify test to allow optional extended plugin Signed-off-by: Craig Perkins <cwperx@amazon.com> * Only optional extended plugins Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add additional warning message Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add to CHANGELOG Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add tag to make extended plugin optional Signed-off-by: Craig Perkins <cwperx@amazon.com> * Only send plugin names when serializing PluginInfo Signed-off-by: Craig Perkins <cwperx@amazon.com> * Keep track of optional extended plugins in separate set Signed-off-by: Craig Perkins <cwperx@amazon.com> * Include in ser/de of PluginInfo Signed-off-by: Craig Perkins <cwperx@amazon.com> * Change to 3_0_0 Signed-off-by: Craig Perkins <cwperx@amazon.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com> Change version in PluginInfo to V_2_19_0 after backport to 2.x merged (opensearch-project#16947) Signed-off-by: Craig Perkins <cwperx@amazon.com> Support object fields in star-tree index (opensearch-project#16728) --------- Signed-off-by: bharath-techie <bharath78910@gmail.com> Bump ch.qos.logback:logback-core from 1.5.12 to 1.5.16 in /test/fixtures/hdfs-fixture (opensearch-project#16951) * Bump ch.qos.logback:logback-core in /test/fixtures/hdfs-fixture Bumps [ch.qos.logback:logback-core](https://github.com/qos-ch/logback) from 1.5.12 to 1.5.16. - [Commits](qos-ch/logback@v_1.5.12...v_1.5.16) --- updated-dependencies: - dependency-name: ch.qos.logback:logback-core dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> [Workload Management] Add Workload Management IT (opensearch-project#16359) * add workload management IT Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> * address comments Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> --------- Signed-off-by: Ruirui Zhang <mariazrr@amazon.com> Add new benchmark config for nested workload (opensearch-project#16956) Signed-off-by: Rishabh Singh <sngri@amazon.com> Bump com.azure:azure-core-http-netty from 1.15.5 to 1.15.7 in /plugins/repository-azure (opensearch-project#16952) * Bump com.azure:azure-core-http-netty in /plugins/repository-azure Bumps [com.azure:azure-core-http-netty](https://github.com/Azure/azure-sdk-for-java) from 1.15.5 to 1.15.7. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-core-http-netty_1.15.5...azure-core-http-netty_1.15.7) --- updated-dependencies: - dependency-name: com.azure:azure-core-http-netty dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Always use constant_score query for match_only_text (opensearch-project#16964) In some cases, when we create a term query over a `match_only_text` field, it may still try to compute scores, which prevents early termination. We should *always* use a constant score query when querying `match_only_text`, since we don't have the statistics required to compute scores. --------- Signed-off-by: Michael Froh <froh@amazon.com> Changes to support unmapped fields in metric aggregation (opensearch-project#16481) Avoids exception when querying unmapped field when star tree experimental feature is enables. --------- Signed-off-by: expani <anijainc@amazon.com> Use async client for delete blob or path in S3 Blob Container (opensearch-project#16788) * Use async client for delete blob or path in S3 Blob Container Signed-off-by: Ashish Singh <ssashish@amazon.com> * Fix UTs Signed-off-by: Ashish Singh <ssashish@amazon.com> * Fix failures in S3BlobStoreRepositoryTests Signed-off-by: Ashish Singh <ssashish@amazon.com> * Fix S3BlobStoreRepositoryTests Signed-off-by: Ashish Singh <ssashish@amazon.com> * Fix failures in S3RepositoryThirdPartyTests Signed-off-by: Ashish Singh <ssashish@amazon.com> * Fix failures in S3RepositoryPluginTests Signed-off-by: Ashish Singh <ssashish@amazon.com> --------- Signed-off-by: Ashish Singh <ssashish@amazon.com> Fix Shallow copy snapshot failures on closed index (opensearch-project#16868) * Fix shallow v1 snapshot failures on closed index Signed-off-by: Shubh Sahu <shubhvs@amazon.com> * UT fix Signed-off-by: Shubh Sahu <shubhvs@amazon.com> * Adding UT Signed-off-by: Shubh Sahu <shubhvs@amazon.com> * small fix Signed-off-by: Shubh Sahu <shubhvs@amazon.com> * Addressing comments Signed-off-by: Shubh Sahu <shubhvs@amazon.com> * Addressing comments Signed-off-by: Shubh Sahu <shubhvs@amazon.com> * Modifying IT to restore snapshot Signed-off-by: Shubh Sahu <shubhvs@amazon.com> --------- Signed-off-by: Shubh Sahu <shubhvs@amazon.com> Co-authored-by: Shubh Sahu <shubhvs@amazon.com> Add Response Status Number in http trace logs. (opensearch-project#16978) Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> support termQueryCaseInsensitive/termQuery can search from doc_value in flat_object/keyword field (opensearch-project#16974) Signed-off-by: kkewwei <kewei.11@bytedance.com> Signed-off-by: kkewwei <kkewwei@163.com> use the correct type to widen the sort fields when merging top docs (opensearch-project#16881) * use the correct type to widen the sort fields when merging top docs Signed-off-by: panguixin <panguixin@bytedance.com> * fix Signed-off-by: panguixin <panguixin@bytedance.com> * apply commments Signed-off-by: panguixin <panguixin@bytedance.com> * changelog Signed-off-by: panguixin <panguixin@bytedance.com> * add more tests Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com> Fix multi-value sort for unsigned long (opensearch-project#16732) * Fix multi-value sort for unsigned long Signed-off-by: panguixin <panguixin@bytedance.com> * Add initial rest-api-spec tests Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * add more rest tests Signed-off-by: panguixin <panguixin@bytedance.com> * fix Signed-off-by: panguixin <panguixin@bytedance.com> * fix Signed-off-by: panguixin <panguixin@bytedance.com> * Extend MultiValueMode with dedicated support of unsigned_long doc values Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Add CHANGELOG.md, minor cleanups Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Correct the license headers Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Correct the @publicapi version Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Replace SingletonSortedNumericUnsignedLongValues with LongToSortedNumericUnsignedLongValues (as per review comments) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> --------- Signed-off-by: panguixin <panguixin@bytedance.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Update Gradle to 8.12 (opensearch-project#16884) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> `phone-search` analyzer: don't emit sip/tel prefix, int'l prefix, extension & unformatted input (opensearch-project#16993) * `phone-search` analyzer: don't emit int'l prefix this was an oversight in the initial implementation: if the tokenizer emits the international calling prefix in the search analyzer then all documents with the same international calling prefix will match. e.g. when searching for `+1-555-123-4567` not only documents with this number would match but also any other document with a `1` token (i.e. any other number with this prefix). thus the search functionality is currently broken for this analyzer, making it useless. the test coverage has now been extended to cover these and other use-cases. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> * `phone-search` analyzer: don't emit extension & unformatted input if these tokens are emitted it meant that phone numbers with other international dialling prefixes still matched. e.g. searching for `+1 1234` would also match a number stored as `+2 1234`, which was wrong. the tokens still need to be emited for the `phone` analyzer, e.g. when the user only enters the extension / local number it should still match, the same is with the other ngrams: these are needed for search-as-you-type style queries where the user input needs to match against partial phone numbers. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> * `phone-search` analyzer: don't emit sip/tel prefix in line with the previous two commits, this is something else the search analyzer shouldn't emit since otherwise searching for any number with such a prefix will match _any_ document with the same prefix. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> --------- Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Limit RW separation to remote store enabled clusters and update recovery flow (opensearch-project#16760) * Update search only replica recovery flow This PR includes multiple changes to search replica recovery. 1. Change search only replica copies to recover as empty store instead of PEER. This will run a store recovery that syncs segments from remote store directly and eliminate any primary communication. 2. Remove search replicas from the in-sync allocation ID set and update routing table to exclude them from allAllocationIds. This ensures primaries aren't tracking or validating the routing table for any search replica's presence. 3. Change search replica validation to require remote store. There are versions of the above changes that are still possible with primary based node-node replication, but I don't think they are worth making at this time. Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * more coverage Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * add changelog entry Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * add assertions that Search Replicas are not in the in-sync id set nor the AllAllocationIds set in the routing table Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * update async task to only run if the FF is enabled and we are a remote store cluster. This check had previously only checked for segrep Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * clean up max shards logic Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * remove search replicas from check during renewPeerRecoveryRetentionLeases Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * Revert "update async task to only run if the FF is enabled and we are a remote store cluster." reverting this, we already check for remote store earlier. This reverts commit 48ca1a3. Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * Add more tests for failover case Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * Update remotestore restore logic and add test ensuring we can restore only writers when red Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * Fix Search replicas to honor node level recovery limits Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * Fix translog UUID mismatch on existing store recovery. This commit adds PR feedback and recovery tests post node restart. Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * Fix spotless Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * Fix bug with remote restore and add more tests Signed-off-by: Marc Handalian <marc.handalian@gmail.com> --------- Signed-off-by: Marc Handalian <marc.handalian@gmail.com> Fix case insensitive and escaped query on wildcard (opensearch-project#16827) * fix case insensitive and escaped query on wildcard Signed-off-by: gesong.samuel <gesong.samuel@bytedance.com> * add changelog Signed-off-by: gesong.samuel <gesong.samuel@bytedance.com> --------- Signed-off-by: gesong.samuel <gesong.samuel@bytedance.com> Signed-off-by: Michael Froh <froh@amazon.com> Co-authored-by: gesong.samuel <gesong.samuel@bytedance.com> Co-authored-by: Michael Froh <froh@amazon.com> Bump opentelemetry from 1.41.0 to 1.46.0 and opentelemetry-semconv from 1.27.0-alpha to 1.29.0-alpha (opensearch-project#17000) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> TransportBulkAction.doRun() (opensearch-project#16950) Signed-off-by: kkewwei <kewei.11@bytedance.com> Signed-off-by: kkewwei <kkewwei@163.com> Show only intersecting buckets to the Adjacency matrix aggregation (opensearch-project#11733) Signed-off-by: Ivan Brusic <ivan@brusic.com> Bump com.google.re2j:re2j from 1.7 to 1.8 in /plugins/repository-hdfs (opensearch-project#17012) * Bump com.google.re2j:re2j from 1.7 to 1.8 in /plugins/repository-hdfs Bumps [com.google.re2j:re2j](https://github.com/google/re2j) from 1.7 to 1.8. - [Release notes](https://github.com/google/re2j/releases) - [Commits](google/re2j@re2j-1.7...re2j-1.8) --- updated-dependencies: - dependency-name: com.google.re2j:re2j dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump com.nimbusds:oauth2-oidc-sdk from 11.20.1 to 11.21 in /plugins/repository-azure (opensearch-project#17010) * Bump com.nimbusds:oauth2-oidc-sdk in /plugins/repository-azure Bumps [com.nimbusds:oauth2-oidc-sdk](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions) from 11.20.1 to 11.21. - [Changelog](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions/src/master/CHANGELOG.txt) - [Commits](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions/branches/compare/11.21..11.20.1) --- updated-dependencies: - dependency-name: com.nimbusds:oauth2-oidc-sdk dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> improve `PhoneNumberAnalyzerTests#testTelPrefixSearch` (opensearch-project#17016) this way we ensure that it doesn't include any additional tokens which we don't want. this is a follow-up to commit 4d94399 / opensearch-project#16993. Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com> Filter shards for sliced search at coordinator (opensearch-project#16771) * Filter shards for sliced search at coordinator Prior to this commit, a sliced search would fan out to every shard, then apply a MatchNoDocsQuery filter on shards that don't correspond to the current slice. This still creates a (useless) search context on each shard for every slice, though. For a long-running sliced scroll, this can quickly exhaust the number of available scroll contexts. This change avoids fanning out to all the shards by checking at the coordinator if a shard is matched by the current slice. This should reduce the number of open scroll contexts to max(numShards, numSlices) instead of numShards * numSlices. --------- Signed-off-by: Michael Froh <froh@amazon.com> Upgrade HttpCore5/HttpClient5 to support ExtendedSocketOption in HttpAsyncClient (opensearch-project#16757) * upgrade httpcore5/httpclient5 to support ExtendedSocketOption in HttpAsyncClient Signed-off-by: kkewwei <kewei.11@bytedance.com> Signed-off-by: kkewwei <kkewwei@163.com> * Use the Upgrade flow by default Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Update Reactor Netty to 1.1.26.Final Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Add SETTING_H2C_MAX_CONTENT_LENGTH to configure h2cMaxContentLength for reactor-netty4 transport Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Update Apache HttpCore5 to 5.3.2 Signed-off-by: Andriy Redko <andriy.redko@aiven.io> --------- Signed-off-by: kkewwei <kewei.11@bytedance.com> Signed-off-by: kkewwei <kkewwei@163.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Update version checks for backport (opensearch-project#17030) Signed-off-by: Michael Froh <froh@amazon.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Michael Froh <froh@amazon.com> Fix versions and breaking API changes (opensearch-project#17031) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Bump com.nimbusds:nimbus-jose-jwt from 9.47 to 10.0.1 in /test/fixtures/hdfs-fixture (opensearch-project#17011) * Bump com.nimbusds:nimbus-jose-jwt in /test/fixtures/hdfs-fixture Bumps [com.nimbusds:nimbus-jose-jwt](https://bitbucket.org/connect2id/nimbus-jose-jwt) from 9.47 to 10.0.1. - [Changelog](https://bitbucket.org/connect2id/nimbus-jose-jwt/src/master/CHANGELOG.txt) - [Commits](https://bitbucket.org/connect2id/nimbus-jose-jwt/branches/compare/10.0.1..9.47) --- updated-dependencies: - dependency-name: com.nimbusds:nimbus-jose-jwt dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Remove user data from logs when not in debug/trace mode (opensearch-project#17007) * Remove user data from logs when not in debug/trace mode Signed-off-by: Mohit Godwani <mgodwan@amazon.com> Remove user data from logs when not in debug/trace mode (opensearch-project#17007) * Remove user data from logs when not in debug/trace mode Signed-off-by: Mohit Godwani <mgodwan@amazon.com> Signed-off-by: meetvm <meetvm@amazon.com>
1 parent 634b84a commit 1c5f519

File tree

9 files changed

+89
-17
lines changed

9 files changed

+89
-17
lines changed

‎modules/reindex/src/yamlRestTest/resources/rest-api-spec/test/delete_by_query/50_wait_for_active_shards.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
match_all: {}
2626

2727
- match:
28-
failures.0.cause.reason: /Not.enough.active.copies.to.meet.shard.count.of.\[4\].\(have.1,.needed.4\)..Timeout\:.\[1s\],.request:.+/
28+
failures.0.cause.reason: /Not.enough.active.copies.to.meet.shard.count.of.\[4\].\(have.1,.needed.4\)..Timeout\:.\[1s\]/
2929

3030
- do:
3131
indices.refresh: {}

‎modules/reindex/src/yamlRestTest/resources/rest-api-spec/test/reindex/60_wait_for_active_shards.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
dest:
2626
index: dest
2727
- match:
28-
failures.0.cause.reason: /Not.enough.active.copies.to.meet.shard.count.of.\[4\].\(have.1,.needed.4\)\..Timeout\:.\[1s\],.request:.+/
28+
failures.0.cause.reason: /Not.enough.active.copies.to.meet.shard.count.of.\[4\].\(have.1,.needed.4\)\..Timeout\:.\[1s\]/
2929

3030
- do:
3131
reindex:

‎modules/reindex/src/yamlRestTest/resources/rest-api-spec/test/update_by_query/50_consistency.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121
wait_for_active_shards: 4
2222
timeout: 1s
2323
- match:
24-
failures.0.cause.reason: /Not.enough.active.copies.to.meet.shard.count.of.\[4\].\(have.1,.needed.4\)..Timeout\:.\[1s\],.request:.+/
24+
failures.0.cause.reason: /Not.enough.active.copies.to.meet.shard.count.of.\[4\].\(have.1,.needed.4\)..Timeout\:.\[1s\]/
2525

2626
- do:
2727
update_by_query:

‎server/src/internalClusterTest/java/org/opensearch/action/support/WaitActiveShardCountIT.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ public void testReplicationWaitsForActiveShardCount() throws Exception {
7676
assertThat(e.status(), equalTo(RestStatus.SERVICE_UNAVAILABLE));
7777
assertThat(
7878
e.getMessage(),
79-
startsWith("[test][0] Not enough active copies to meet shard count of [2] (have 1, needed 2). Timeout: [100ms], request:")
79+
startsWith("[test][0] Not enough active copies to meet shard count of [2] (have 1, needed 2). Timeout: [100ms]")
8080
);
8181
// but really, all is well
8282
}
@@ -120,7 +120,7 @@ public void testReplicationWaitsForActiveShardCount() throws Exception {
120120
startsWith(
121121
"[test][0] Not enough active copies to meet shard count of ["
122122
+ ActiveShardCount.ALL
123-
+ "] (have 2, needed 3). Timeout: [100ms], request:"
123+
+ "] (have 2, needed 3). Timeout: [100ms]"
124124
)
125125
);
126126
// but really, all is well

‎server/src/main/java/org/opensearch/action/support/replication/ReplicationOperation.java

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -141,15 +141,7 @@ public void execute() throws Exception {
141141
final ShardRouting primaryRouting = primary.routingEntry();
142142
final ShardId primaryId = primaryRouting.shardId();
143143
if (activeShardCountFailure != null) {
144-
finishAsFailed(
145-
new UnavailableShardsException(
146-
primaryId,
147-
"{} Timeout: [{}], request: [{}]",
148-
activeShardCountFailure,
149-
request.timeout(),
150-
request
151-
)
152-
);
144+
finishAsFailed(new UnavailableShardsException(primaryId, "{} Timeout: [{}]", activeShardCountFailure, request.timeout()));
153145
return;
154146
}
155147

‎server/src/main/java/org/opensearch/action/support/replication/TransportReplicationAction.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1246,7 +1246,7 @@ void finishOnSuccess(Response response) {
12461246
}
12471247

12481248
void retryBecauseUnavailable(ShardId shardId, String message) {
1249-
retry(new UnavailableShardsException(shardId, "{} Timeout: [{}], request: [{}]", message, request.timeout(), request));
1249+
retry(new UnavailableShardsException(shardId, "{} Timeout: [{}]", message, request.timeout()));
12501250
}
12511251
}
12521252

‎server/src/main/java/org/opensearch/action/update/UpdateHelper.java

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@
5858
import org.opensearch.index.shard.IndexShard;
5959
import org.opensearch.script.Script;
6060
import org.opensearch.script.ScriptService;
61+
import org.opensearch.script.ScriptType;
6162
import org.opensearch.script.UpdateScript;
6263
import org.opensearch.search.lookup.SourceLookup;
6364

@@ -128,7 +129,11 @@ Tuple<UpdateOpType, Map<String, Object>> executeScriptedUpsert(Map<String, Objec
128129

129130
if (operation != UpdateOpType.CREATE && operation != UpdateOpType.NONE) {
130131
// Only valid options for an upsert script are "create" (the default) or "none", meaning abort upsert
131-
logger.warn("Invalid upsert operation [{}] for script [{}], doing nothing...", operation, script.getIdOrCode());
132+
if (logger.isDebugEnabled() || ScriptType.STORED.equals(script.getType())) {
133+
logger.warn("Invalid upsert operation [{}] for script [{}], doing nothing...", operation, script.getIdOrCode());
134+
} else {
135+
logger.warn("Invalid upsert operation [{}] for given script", operation);
136+
}
132137
operation = UpdateOpType.NONE;
133138
}
134139

‎server/src/main/java/org/opensearch/index/translog/transfer/TranslogTransferManager.java

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -206,7 +206,8 @@ public boolean transferSnapshot(TransferSnapshot transferSnapshot, TranslogTrans
206206
} catch (Exception ex) {
207207
logger.error(() -> new ParameterizedMessage("Transfer failed for snapshot {}", transferSnapshot), ex);
208208
captureStatsOnUploadFailure();
209-
translogTransferListener.onUploadFailed(transferSnapshot, ex);
209+
Exception exWithoutSuppressed = new TranslogUploadFailedException(ex.getMessage());
210+
translogTransferListener.onUploadFailed(transferSnapshot, exWithoutSuppressed);
210211
return false;
211212
}
212213
}

‎server/src/test/java/org/opensearch/index/translog/transfer/TranslogTransferManagerTests.java

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,80 @@ public void onUploadFailed(TransferSnapshot transferSnapshot, Exception ex) {
206206
assertEquals(4, fileTransferTracker.allUploaded().size());
207207
}
208208

209+
public void testTransferSnapshotOnFileTransferUploadFail() throws Exception {
210+
AtomicInteger fileTransferSucceeded = new AtomicInteger();
211+
AtomicInteger fileTransferFailed = new AtomicInteger();
212+
AtomicInteger translogTransferSucceeded = new AtomicInteger();
213+
AtomicInteger translogTransferFailed = new AtomicInteger();
214+
215+
doAnswer(invocationOnMock -> {
216+
ActionListener<TransferFileSnapshot> listener = (ActionListener<TransferFileSnapshot>) invocationOnMock.getArguments()[2];
217+
Set<TransferFileSnapshot> transferFileSnapshots = (Set<TransferFileSnapshot>) invocationOnMock.getArguments()[0];
218+
219+
TransferFileSnapshot actualFileSnapshot = transferFileSnapshots.iterator().next();
220+
FileTransferException testException = new FileTransferException(
221+
actualFileSnapshot,
222+
new RuntimeException("FileTransferUploadNeedsToFail-Exception")
223+
);
224+
225+
listener.onFailure(testException);
226+
transferFileSnapshots.stream().skip(1).forEach(listener::onResponse);
227+
return null;
228+
}).when(transferService).uploadBlobs(anySet(), anyMap(), any(ActionListener.class), any(WritePriority.class));
229+
230+
FileTransferTracker fileTransferTracker = new FileTransferTracker(
231+
new ShardId("index", "indexUUid", 0),
232+
remoteTranslogTransferTracker
233+
) {
234+
@Override
235+
public void onSuccess(TransferFileSnapshot fileSnapshot) {
236+
fileTransferSucceeded.incrementAndGet();
237+
super.onSuccess(fileSnapshot);
238+
}
239+
240+
@Override
241+
public void onFailure(TransferFileSnapshot fileSnapshot, Exception e) {
242+
fileTransferFailed.incrementAndGet();
243+
super.onFailure(fileSnapshot, e);
244+
}
245+
};
246+
247+
TranslogTransferManager translogTransferManager = new TranslogTransferManager(
248+
shardId,
249+
transferService,
250+
remoteBaseTransferPath.add(TRANSLOG.getName()),
251+
remoteBaseTransferPath.add(METADATA.getName()),
252+
fileTransferTracker,
253+
remoteTranslogTransferTracker,
254+
DefaultRemoteStoreSettings.INSTANCE,
255+
isTranslogMetadataEnabled
256+
);
257+
258+
SetOnce<Exception> exception = new SetOnce<>();
259+
assertFalse(translogTransferManager.transferSnapshot(createTransferSnapshot(), new TranslogTransferListener() {
260+
@Override
261+
public void onUploadComplete(TransferSnapshot transferSnapshot) {
262+
translogTransferSucceeded.incrementAndGet();
263+
}
264+
265+
@Override
266+
public void onUploadFailed(TransferSnapshot transferSnapshot, Exception ex) {
267+
translogTransferFailed.incrementAndGet();
268+
exception.set(ex);
269+
}
270+
}));
271+
272+
assertNotNull(exception.get());
273+
assertTrue(exception.get() instanceof TranslogUploadFailedException);
274+
assertEquals("Failed to upload 1 files during transfer", exception.get().getMessage());
275+
assertEquals(0, exception.get().getSuppressed().length);
276+
assertEquals(3, fileTransferSucceeded.get());
277+
assertEquals(1, fileTransferFailed.get());
278+
assertEquals(0, translogTransferSucceeded.get());
279+
assertEquals(1, translogTransferFailed.get());
280+
assertEquals(3, fileTransferTracker.allUploaded().size());
281+
}
282+
209283
public void testTransferSnapshotOnUploadTimeout() throws Exception {
210284
doAnswer(invocationOnMock -> {
211285
Set<TransferFileSnapshot> transferFileSnapshots = invocationOnMock.getArgument(0);

0 commit comments

Comments
 (0)