Terraform not allowing to pass nested values (like assume_role.role_arn) to the backend.s3 configuration using cli (-backend-config="KEY=VALUE") and config file (-backend-config=PATH)

[Humeid-Ussene-Jocordasse]

Terraform Version

1.11.4

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.33.0"
    }
  }
  
  backend "s3" {
    bucket = ""
    use_lockfile=""
    key=""
    region=""
    assume_role = {
      role_arn = ""
      external_id = ""
    }
  }
    required_version = ">= 0.14.9"
}
...terraform config...

Debug Output

not applicable

Expected Behavior

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Actual Behavior

Initializing the backend...
╷
│ Error: Invalid backend configuration argument
│ 
│ The backend configuration argument "assume_role.role_arn" given on the command line is not expected for the selected backend type.
╵
╷
│ Error: Invalid backend configuration argument
│ 
│ The backend configuration argument "assume_role.external_id" given on the command line is not expected for the selected backend type.

Steps to Reproduce

terraform -chdir=.iac init  \
      -backend=true \
      -backend-config="bucket=my-humble-bucket" \
      -backend-config="key=my-humble-key.tfstate" \
      -backend-config="region=af-south-1" \
      -backend-config="use_lockfile=true" \
      -backend-config="assume_role.role_arn=arn:aws:iam::123456789012:role/my-humble-role" \
      -backend-config="assume_role.external_id=my-humble-id" \
      -force-copy

Additional Context

I'm having the same issue when i true to use those exact same values on backend.config file, trying to pass with the -backend-config=PATH flag...

I noticed this issue after upgrading the Terraform version in our team's Continuous Integration tool from v1.8.5 to v1.11.4. Previously, we were able to provide the role_arn programmatically because it was a top-level attribute in the backend.s3 block. Now that it has to be nested under assume_role, Terraform throws an error when we try to pass it dynamically.

Edit:
I've found out this still works when using a .hcl file, passing a value like this one:

bucket = "my-humble-bucket"
key="test/my-humble-key.tfstate"
region="af-south-1"
use_lockfile="true"
assume_role= {
    role_arn = "arn:aws:iam::123456789012:role/my-humble-role"
    external_id="my-humble-id"
}

References

No response

Generative AI / LLM assisted development?

No response

[crw]

Thanks for this report!

[apparentlymart]

In case it's somehow useful, a long time ago I wrote github.com/apparentlymart/go-hcl-overlay with the goal of supporting stuff like this where a system needs to merge some command line options in with existing HCL body content.

Even if that library is not directly useful as an import, I remember there being some interesting design tradeoffs in making that work which might give some ideas on similar tradeoffs to make in solving this for Terraform. In particular, I experimented with a few different ways to deal with a situation where e.g. the CLI says assume_role.role_arn but yet there isn't already an assume_role block in the configuration for that to override into, and picked one that seemed to work relatively well for reasonable cases like this.

The example I included in the repository uses the very opinionated ExtractCLIOptions whose opinions don't match how -backend-config already works, but I think ParseCLIArgument is closer to how Terraform wants to treat -backend-config where the key=value is given as the value of an option rather than as the option itself.

Offered just in the hope that it's helpful!

[SarahFrench]

Hi @apparentlymart , thanks for sharing that project! I had a go using it directly but it looks like attributes with NestedType set cannot be handled currently (I get Invalid argument errors for assume_role.role_arn etc. from cli_args.go, due to it being an attribute where the path contains >1 identifier). I like the overlay approach though, and will keep investigating, as this is a good learning opportunity around cty and hcl.

[apparentlymart]

Ahh hmm yes I suppose I wrote this long enough ago that the idea of using the attribute syntax in a block-like way hadn't happened yet, and so my old implementation doesn't support that... it only wants to allow dotting through block types and their labels.

"Overlaying" into attributes is actually likely to be somewhat harder than what I implemented there previously because from HCL's perspective the given expression hasn't been evaluated yet and so it doesn't have any idea what type of value is likely to appear there until someone eventually calls Value on the expression.

So following a similar implementation approach as what I was using so far would probably mean adding a new wrapping implementation of hcl.Expression, similar to how it already has a wrapping implementation of hcl.Body, and then using that wrapper expression type in the returned attribute whenever the given attribute path has more than one segment, so that the wrapper expression's Value method can try to handle this as a merger of two cty.Value instead of as a merger of two hcl.Body. 😬

If it's going to ultimately end up having to do the work in the cty.Value realm anyway then probably better to try to apply the "overlays" after HCL has already done its work, directly against the result cty.Value, so that nested blocks and structural attributes can be treated in the same way, just as cty objects.

I'm sorry that it wasn't more helpful!