Skip to content

Enabling Support for Conditional Single-Part Upload [AWS S3] #18092

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

x-INFiN1TY-x
Copy link

Description

Adds executeSingleUploadIfEtagMatches(...) to S3BlobStore, enabling support for conditional single‑part upload via the S3 If-Match header.

  • **Success **: invokes onResponse with the new ETag.
  • Precondition Failed (412): invokes onFailure with an OpenSearchException("stale_primary_shard").
  • Other S3/SDK errors: wrapped in IOException and passed to onFailure.

Diag

Test Coverage

  1. Precondition Failed (HTTP 412)
    Setup: Mock S3Exception 412
    Expectation:

    • Throws IOException("Unable to upload object [<blob>] due to ETag mismatch")
    • Listener fails with OpenSearchException("stale_primary_shard")
    • onResponse never called
  2. General S3Exception (HTTP 403)
    Setup: Mock S3Exception 403
    Expectation:

    • Throws IOException("S3 error during upload [<blob>]: Access Denied") (cause is the original S3Exception)
    • Listener’s onFailure invoked (message contains blob name)
    • onResponse never called
  3. Successful Upload
    Setup: Mock PutObjectResponse with non-null ETag
    Expectation:

    • Listener’s onResponse receives that ETag
    • All request fields (bucket, key, contentLength, storageClass, ACL, metadata, ifMatch, SSE) are correct
    • onFailure never called
  4. SDK Exception
    Setup: Mock SdkException
    Expectation:

    • Throws IOException("S3 upload failed for [<blob>]", sdkException)
    • Listener’s onFailure receives that IOException
  5. Null ETag in Response
    Setup: Mock PutObjectResponse with eTag == null
    Expectation:

    • Throws IOException("S3 upload for [<blob>] returned null ETag…")
    • Listener’s onFailure invoked
    • onResponse never called
  6. Null or Empty Input/ETag

    • Null ETag, non-empty blob: upload succeeds → onResponse gets returned ETag
    • Empty ETag "", zero-length blob: upload succeeds → onResponse gets returned ETag
  7. Blob Size Limits

    • > 5 GB: Throws IllegalArgumentException("Upload request size [<size>] can't be larger than 5gb")
    • > bufferSize: Throws IllegalArgumentException before any S3 interaction

Related Issue

Concerned RFC : RFC #17763
Parent Meta Issue : Meta #17859

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@x-INFiN1TY-x x-INFiN1TY-x changed the title Pr branch conditional single upload Support Conditional Writes for Single-Part S3 Uploads Apr 28, 2025
Copy link
Contributor

❌ Gradle check result for f1c411e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@x-INFiN1TY-x
Copy link
Author

Flaky Test : : #14509

  • "testRecoverWhileUnderLoadAllocateReplicasRelocatePrimariesTest {p0={"cluster.indices.replication.strategy":"DOCUMENT"}} – org.opensearch.recovery.RecoveryWhileUnderLoadIT"

Signed-off-by: Tanishq Ranjan <tqranjan@amazon.com>
@x-INFiN1TY-x x-INFiN1TY-x force-pushed the pr-branch-conditionalSingleUpload branch from e38f1fd to 3951775 Compare April 28, 2025 20:11
Copy link
Contributor

✅ Gradle check result for 3951775: SUCCESS

Copy link

codecov bot commented Apr 28, 2025

Codecov Report

Attention: Patch coverage is 95.23810% with 2 lines in your changes missing coverage. Please review.

Project coverage is 72.57%. Comparing base (d3eb8fe) to head (3951775).
Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
...rg/opensearch/repositories/s3/S3BlobContainer.java 95.23% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #18092      +/-   ##
============================================
+ Coverage     72.53%   72.57%   +0.03%     
- Complexity    67132    67192      +60     
============================================
  Files          5474     5474              
  Lines        310173   310216      +43     
  Branches      45066    45072       +6     
============================================
+ Hits         224993   225131     +138     
+ Misses        66840    66743      -97     
- Partials      18340    18342       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@x-INFiN1TY-x x-INFiN1TY-x changed the title Support Conditional Writes for Single-Part S3 Uploads Enabling Support for Conditional Single-Part Upload [AWS S3] Apr 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants