Pushing state to Hetzner S3

[PlexusisSolaris]

Hello,
so I am trying to push created state into S3 bucket in Hetzner, but I keep getting:

│ Error: Failed to save state
│ 
│ Error saving state: failed to upload state: operation error S3: PutObject, https response error
│ StatusCode: 400, RequestID: tx0000068690c1e6f92cfa8-0067e47576-a37e017-fsn1-prod1-ceph3, HostID:
│ a37e017-fsn1-prod1-ceph3-fsn1-prod1, api error XAmzContentSHA256Mismatch: UnknownError

Has somebody encountered similar error while uploading to S3? Have you uploaded state to S3 while using this module or this is just problem in general with terraform and Hetzner?
Here is a relevant part of debug:

2025-03-26T21:21:19.702+0100 [INFO]  backend-s3: Uploading remote state: tf_backend.operation=Put tf_backend.req_id=b533df7e-9f73-706e-e116-cea72cc9eb34 tf_backend.s3.bucket=talos tf_backend.s3.path=terraform.tfstate
2025-03-26T21:21:19.704+0100 [DEBUG] backend-s3: HTTP Request Sent: aws.region=fsn1 aws.s3.bucket=talos aws.s3.key=terraform.tfstate rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Put tf_backend.req_id=b533df7e-9f73-706e-e116-cea72cc9eb34 tf_backend.s3.bucket=talos tf_backend.s3.path=terraform.tfstate http.request.header.content_type=application/json http.request.header.authorization="AWS4-HMAC-SHA256 Credential=YRYEUEF43S1FLJVCOBDC/20250326/fsn1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-trailer, Signature=*****" http.request.header.x_amz_date=20250326T202119Z http.request.header.x_amz_content_sha256=STREAMING-UNSIGNED-PAYLOAD-TRAILER http.request_content_length=949885 http.request.header.x_amz_trailer=x-amz-checksum-crc32 http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.content_encoding=aws-chunked http.request.header.x_amz_decoded_content_length=949840 http.method=PUT http.url=https://fsn1.your-objectstorage.com/talos/terraform.tfstate?x-id=PutObject net.peer.name=fsn1.your-objectstorage.com http.request.header.accept_encoding=identity http.request.header.amz_sdk_invocation_id=7b468d9c-28de-4e4e-8530-f4f4e92e114c http.request.body="[Redacted: 927.6 KB (949,885 bytes), Type: application/json]" http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.11.3 (+https://www.terraform.io) aws-sdk-go-v2/1.36.0 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/s3#1.75.2 ft/s3-transfer m/G,Z"
2025-03-26T21:21:20.205+0100 [DEBUG] backend-s3: HTTP Response Received: aws.region=fsn1 aws.s3.bucket=talos aws.s3.key=terraform.tfstate rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Put tf_backend.req_id=b533df7e-9f73-706e-e116-cea72cc9eb34 tf_backend.s3.bucket=talos tf_backend.s3.path=terraform.tfstate http.status_code=400 http.response.header.strict_transport_security=max-age=63072000 http.response.header.x_debug_bucket=talos http.response.header.x_amz_request_id=tx00000837b4aa718d95bae-0067e461bf-a622973-fsn1-prod1-ceph3 http.response.header.accept_ranges=bytes http.response.header.content_type=application/xml
  http.response.body=
  | <?xml version="1.0" encoding="UTF-8"?><Error><Code>XAmzContentSHA256Mismatch</Code><Message></Message><BucketName>talos</BucketName><RequestId>tx00000837b4aa718d95bae-0067e461bf-a622973-fsn1-prod1-ceph3</RequestId><HostId>a622973-fsn1-prod1-ceph3-fsn1-prod1</HostId></Error>
   http.duration=500 http.response_content_length=274 http.response.header.date="Wed, 26 Mar 2025 20:21:20 GMT"
2025-03-26T21:21:20.205+0100 [DEBUG] backend-s3: request failed with unretryable error https response error StatusCode: 400, RequestID: tx00000837b4aa718d95bae-0067e461bf-a622973-fsn1-prod1-ceph3, HostID: a622973-fsn1-prod1-ceph3-fsn1-prod1, api error XAmzContentSHA256Mismatch: UnknownError: aws.region=fsn1 aws.s3.bucket=talos aws.s3.key=terraform.tfstate rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Put tf_backend.req_id=b533df7e-9f73-706e-e116-cea72cc9eb34 tf_backend.s3.bucket=talos tf_backend.s3.path=terraform.tfstate

and this is my backend:

terraform {
  backend "s3" {
    bucket = "talos"
    key = "terraform.tfstate"
    skip_region_validation = true
    region = "fsn1"
    endpoints = {
      s3 = "https://fsn1.your-objectstorage.com"
    }
    skip_credentials_validation = true
    skip_requesting_account_id = true
    skip_metadata_api_check = true
    encrypt = false
    use_path_style = true
    skip_s3_checksum = true
    acl = "private"
    workspace_key_prefix = "state"
  }
}

as seen in logs, it is registering correctly env variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, bucket is also working as I have already pushed many logs and metrics into it, any help would be appreciated 🙏

[mrclrchtr]

Basically, this has nothing to do with this module.

Unfortunately, I don't notice any errors in the config. I would ask Hetzner directly how this works or what the problem is.

Of course it could help others if you post the solution here.

[PlexusisSolaris]

Hey @mrclrchtr,
did some digging and Hetzner S3 is incompatible.
AWS SDKs were updated only after the 1.11.1 version of TF, so using version 1.11.1 should be fine(I tested only 1.11.1).

[mrclrchtr]

Thanks for the information