Writable warm replica replication/recovery

[skumawat2025]

Description

This PR aims to implement the Replica Replication and Recovery flow for indices with partial store locality (also known as writable warm indices), which were introduced with the composite directory in PR #12782. It builds upon and addresses comments received on the pending PR #14670.
For writable indices, where primary shards now consist of both localDirectory and remoteDirectory, we've implemented file uploads to remote storage and maintain an LRU fileCache on the node instead of storing all files locally.
We've made several changes to the replication and recovery processes:

1. During replication events on replicas, we now only update the NRTReplicationReaderManager with the latest CheckpointInfoResponse, avoiding unnecessary downloads of actual file diffs on replica.
2. For index close, we are skipping doing a flush on replica shards as during re-open we will anyway sync from remote store.
3. During remote store sync in recovery scenarios, we are skipping coping the actually segment files from remote storage to local and create a new commit with the latest commit info from remote segment metadata file.

Related Issues

Resolves #13647

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 6fde641: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 1963b58: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 3ce8970: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[skumawat2025]

❌ Gradle check result for 1963b58: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Failing tests are flaky: #17364, #15806, #16145

[github-actions]

❌ Gradle check result for cf1b64b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for d920d71: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 17b43da: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for b216afc: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 36.17021% with 30 lines in your changes missing coverage. Please review.

Project coverage is 72.41%. Comparing base (82bbdfb) to head (b216afc).
Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
...earch/index/store/remote/utils/cache/LRUCache.java	0.00%	10 Missing ⚠️
...org/opensearch/index/store/CompositeDirectory.java	65.00%	4 Missing and 3 partials ⚠️
...index/store/remote/utils/cache/SegmentedCache.java	0.00%	4 Missing ⚠️
.../opensearch/index/engine/NRTReplicationEngine.java	25.00%	1 Missing and 2 partials ⚠️
...ing/allocation/allocator/RemoteShardsBalancer.java	0.00%	2 Missing ⚠️
...in/java/org/opensearch/index/shard/IndexShard.java	33.33%	1 Missing and 1 partial ⚠️
.../indices/replication/SegmentReplicationTarget.java	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17390      +/-   ##
============================================
- Coverage     72.59%   72.41%   -0.18%     
+ Complexity    65798    65727      -71     
============================================
  Files          5311     5311              
  Lines        304888   304924      +36     
  Branches      44212    44225      +13     
============================================
- Hits         221323   220819     -504     
- Misses        65501    66021     +520     
- Partials      18064    18084      +20     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.