Virtual-host-style S3 requests fail when using a single endpoint for all services (i.e., AWS_ENDPOINT_URL)

[joe4dev]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

S3 API calls using virtual-host-style requests (default in most SDKs except for botocore) fail when using a single endpoint URL (e.g., AWS_ENDPOINT_URL=http://localhost.localstack.cloud:4566 or AWS_ENDPOINT_URL=http://localhost:4566). This breaks native LocalStack compatibility of many AWS tools such as the AWS Toolkit (aws/aws-toolkit-vscode#2007) and AWS CDK (related to aws/aws-cdk#21014).

AWS endpoint standard: https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-s3@3.830.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v22.14.0

Reproduction Steps

1. Install and start LocalStack: https://docs.localstack.cloud/getting-started/installation/

2. npm install @aws-sdk/client-s3@3.830.0

3. Run node test.js using the following script:

   import { S3Client, CreateBucketCommand, PutObjectCommand} from "@aws-sdk/client-s3";
   
   const BUCKET_NAME = "my-unique-localstack-bucket-name";
   const OBJECT_KEY = "HelloWorld.txt";
   const CONTENT = "HelloWorld!";
   
   const s3 = new S3Client({
       region: "us-east-1",
       endpoint: "http://localhost:4566",
       credentials: {
         accessKeyId: "test",
         secretAccessKey: "test",
       },
       maxAttempts: 1,
   });
   
   const createBucketResponse = await s3.send(new CreateBucketCommand({ Bucket: BUCKET_NAME }));
   const putObjectResponse = await s3.send(new PutObjectCommand({
       Bucket: BUCKET_NAME,
       Key: OBJECT_KEY,
       Body: CONTENT,
       ContentType: "text/plain",
   }));

Link reproducer test suite (including botocore comparison): https://github.com/joe4dev/s3-endpoint-url-testing

Observed Behavior

The JavaScript SDK raises an S3ServiceException [InternalError]: exception while calling s3 with unknown operation: Unable to find operation for request to service s3: PUT
The root cause of this confusing ProtocolParserError is that emulators such as LocalStack cannot reliably detect virtual host-styled S3 requests without the .s3 prefix when using a single global endpoint. Therefore, the PutBucket (or any virtual-host-style request) fails with an internal server error (i.e., it gets interpreted as an XML-style CreateBucket request).

/Users/joe/Downloads/s3_test/node_modules/@smithy/smithy-client/dist-cjs/index.js:388
  const response = new exceptionCtor({
                   ^

S3ServiceException [InternalError]: exception while calling s3 with unknown operation: Unable to find operation for request to service s3: PUT /
    at throwDefaultError (/Users/joe/Downloads/s3_test/node_modules/@smithy/smithy-client/dist-cjs/index.js:388:20)
    at /Users/joe/Downloads/s3_test/node_modules/@smithy/smithy-client/dist-cjs/index.js:397:5
    at de_CommandError (/Users/joe/Downloads/s3_test/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4953:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async /Users/joe/Downloads/s3_test/node_modules/@smithy/middleware-serde/dist-cjs/index.js:36:20
    at async /Users/joe/Downloads/s3_test/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:484:18
    at async /Users/joe/Downloads/s3_test/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /Users/joe/Downloads/s3_test/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:110:22
    at async /Users/joe/Downloads/s3_test/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:137:14
    at async /Users/joe/Downloads/s3_test/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22 {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 500,
    requestId: 'a142a92d-7db0-497f-bf8f-11011f69899f',
    extendedRequestId: 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=',
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Code: 'InternalError',
  RequestId: 'a142a92d-7db0-497f-bf8f-11011f69899f'
}

Node.js v22.14.0

Expected Behavior

S3 requests should work properly when using a single endpoint (e.g., AWS_ENDPOINT_URL=http://localhost:4566), following the AWS standard for endpoint URLs: https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html

The S3 client in the JavaScript SDK should detect that a single endpoint is incompatible with virtual-host-style requests and switch to path-style.
Testing virtual-host-style should still be possible with emulators when using a compatible service-specific endpoint URL such as AWS_ENDPOINT_URL_S3=http://s3.localhost.localstack.cloud:4566

Possible Solution

Link PR: #7137

The Python library botocore could serve as a reference implementation, as demonstrated in the PR#2785 Endpoint resolution v2.0.

The relevant change happens in compute_endpoint_resolver_builtin_defaults where this conditional adjusts force_path_style if a custom client_endpoint_url is used and the endpoint does not match the S3 Accelerate endpoint scheme (only applicable to amazonaws.com endpoints). It forces path-style S3 requests unless the S3 addressing_style configuration explicitly demands virtual-style:

elif client_endpoint_url is not None and not is_s3_accelerate_url(
            client_endpoint_url
        ):
    force_path_style = s3_config.get('addressing_style') != 'virtual'

This change enables S3 APIs to work with a global endpoint (e.g., AWS_ENDPOINT_URL=http://localhost.localstack.cloud) using path-style S3 requests while still allowing to force virtual-style S3 requests if demanded (e.g., when manually setting AWS_ENDPOINT_URL_S3=http://s3.localhost.localstack.cloud:4566 and s3 > addressing_style=virtual).

The solution from botocore is not directly applicable here because the JavaScript SDK offers a boolean-named configuration forcePathStyle=true|false (auto by default) instead of the triple-choice option addressing_style=auto|virtual|path. We should allow for virtual-style-requests, either by interpreting forcePathStyle=false (if auto reliably detect undefined) as addressing_style_=virtual or by detecting compatible subdomains starting with s3. (e.g., http://s3.localhost.localstack.cloud:4566).

Additional Information/Context

This issue may be related to Smithy code generation and might be solvable using the Smithy rules engine
Related example from botocore for S3 endpoint ruleset: https://github.com/boto/botocore/blob/d1fd992119b5df4f4d2169e2383ab99288466e8b/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json

Disclaimer: I work for LocalStack

[kuhe]

Do you currently instruct LocalStack users to set forcePathStyle on the S3Client instance constructor under applicable conditions?