[Bug]: tags_all applied incorrectly on aws_opensearchserverless_collection

[albferrari]

Terraform Core Version

1.7.2

AWS Provider Version

5.75.0

Affected Resource(s)

aws_opensearchserverless_collection

Expected Behavior

the resource should be created

Actual Behavior

the resource creation failed

Relevant Error/Panic Output Snippet

╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for
│ module.micro_svc.module.buyer_account_search[0].aws_opensearchserverless_collection.os_collection
│ to include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .tags_all: new element "Mode" has appeared.

Terraform Configuration Files

provider "aws" {
  region = "us-east-1"

  default_tags {
    tags = {
      Environment  = "dev"
      Mode         = "dev"
      "mt-born-on" = timestamp()
    }
  }
  ignore_tags {
    keys         = ["mt-born-on"]
    key_prefixes = ["kubernetes.io/"]
  }
}
resource "aws_opensearchserverless_collection" "os_collection" {
  name = "test"
  type       = "SEARCH"
}

Steps to Reproduce

terraform init
terraform apply

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[apparentlymart]

I have another reproduction case for this, and also some theories on what is causing it.

Here's my test configuration:

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
    terraform = {
      source = "terraform.io/builtin/terraform"
    }
  }
}

resource "terraform_data" "placeholder" {
}

provider "aws" {
  region = "us-west-2"

  default_tags {
    tags = {
        "Problem" = terraform_data.placeholder.id
    }
  }
}

resource "aws_vpc" "example" {
  cidr_block = "192.168.4.0/24"

  tags = {
    "Another" = "no problem"
  }
}

Notice that terraform_data.placeholder.id would be unknown when terraform_data.placeholder is planned for creation, and so I'm effectively setting the default tag named Problem to an unknown value. That creates essentially the same situation as setting a default tag to timestamp() in the original report's example, but I wrote it this way because it's closer to a real example: using a not-yet-finalized attribute from some other resource to populate a tag is arguably a more reasonable thing to do than using a constantly-changing timestamp. Nonetheless, I believe the problem is fundamentally about the value being unknown, regardless of why it's unknown.

When I plan/apply the above configuration with valid AWS credentials, I get a similar error as the original report:

$ ./terraform apply

Terraform used the selected providers to generate the following execution plan.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_vpc.example will be created
  + resource "aws_vpc" "example" {
      + arn                                  = (known after apply)
      + cidr_block                           = "192.168.4.0/24"
      + default_network_acl_id               = (known after apply)
      + default_route_table_id               = (known after apply)
      + default_security_group_id            = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_dns_hostnames                 = (known after apply)
      + enable_dns_support                   = true
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = "default"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + ipv6_cidr_block_network_border_group = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + tags                                 = {
          + "Another" = "no problem"
        }
      + tags_all                             = {
          + "Another" = "no problem"
        }
    }

  # terraform_data.placeholder will be created
  + resource "terraform_data" "placeholder" {
      + id = (known after apply)
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

terraform_data.placeholder: Creating...
terraform_data.placeholder: Creation complete after 0s [id=aac8f555-befe-1cf6-422e-c75ffe817aee]
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_vpc.example to include new values learned so far during apply,
│ provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .tags_all:
│ new element "Problem" has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵

I think the general problem here is that the provider is treating an unknown value in default_tags as if it isn't set at all, and so the initial plan claims that tags_all will not include the additional tag. I suspect that in practice this problem originates in the legacy Terraform SDK that these resource types are using rather than directly in the provider: this seems to be a variation of the classic problem that the legacy SDK treats unknown provider configuration arguments as unset, preventing the provider from detecting that they were unknown and reacting to that.

I would suggest that the correct behavior would be for the provider to either plan for tags_all to be wholly unknown, or (ideally) to plan for just the one unknown tag value to be unknown, like this:

      + tags_all                             = (known after apply)

or

      + tags_all                             = {
          + "Another" = "no problem"
          + "Problem" = (known after apply)
        }

That would then allow the provider to include the final known value for the tag in the final plan (created during the apply phase) without violating Terraform Core's consistency rules.

Of course, if the SDK is the root cause of this then that'd be easier said than done, particularly for the default_tags feature which is cross-cutting across a large number of resource types. 😖