[Bug]: aws_cloudwatch_metric_alarm: Inconsistent plan when depending on data resource

[dennis-grunert]

Terraform Core Version

1.6.4

AWS Provider Version

5.28.0

Affected Resource(s)

aws_cloudwatch_metric_alarm

Expected Behavior

When following the README.md of the minimal example repository, the apply should happen without any error.

Actual Behavior

The apply fails with the following error:

Error: Provider produced inconsistent final plan

When expanding the plan for aws_cloudwatch_metric_alarm.loadbalancer_high5xx to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .metric_query: planned set element
cty.ObjectVal(map[string]cty.Value{"account_id":cty.StringVal(""), "expression":cty.StringVal(""), "id":cty.StringVal("m1"), "label":cty.StringVal(""), "metric":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"dimensions":cty.UnknownVal(cty.Map(cty.String)),
"metric_name":cty.StringVal("HTTPCode_Target_5XX_Count"), "namespace":cty.StringVal("AWS/ApplicationELB"), "period":cty.NumberIntVal(60), "stat":cty.StringVal("Sum"), "unit":cty.StringVal("Count")})}), "period":cty.NullVal(cty.Number), "return_data":cty.True}) does not correlate
with any element in actual.

This is a bug in the provider, which should be reported in the provider's own issue tracker.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

A minimal example to reproduce the inconsistent plan is available here.

Steps to Reproduce

Follow the instructions of the README.md of the above repository to reproduce the issue.

Debug Output

No response

Panic Output

No response

Important Factoids

The minimal example creates an Beanstalk Environment. With a data.aws_lb resource, the (application) load balancer belonging to the Beanstalk is extracted and used in a CloudWatch alarm.

When a setting in the Beanstalk that has nothing to do with the load balancer is changed later, an inconsistent plan for the CloudWatch alarm is shown.

Due to the nature of the Beanstalk being created with a CloudFormation stack, the ARN of the load balancer is only known to the data resource at the apply phase. But it is unclear why an inconsistent plan is created when the ARN of the load balancer does not change in the end.

References

Minimal example to reproduce: https://github.com/dennis-grunert/minimal-example-beanstalk

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[cunhafinrix]

We are having the same problem. We created this alarm

resource "aws_cloudwatch_metric_alarm" "alb_4xx_error_rate" {
  alarm_name          = "HTTP 4xx error %"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  alarm_description   = "Client 4xx errors breached 75% for sustained period."
  treat_missing_data  = "notBreaching"
  evaluation_periods  = 5
  threshold           = 70
  alarm_actions       = [data.aws_sns_topic.alarm_topic.arn]

  metric_query {
    id          = "e1"
    expression  = "m2/m1*100"
    label       = "Error Rate"
    return_data = "true"
  }

  metric_query {
    id = "m1"

    metric {
      metric_name = "RequestCount"
      namespace   = "AWS/ApplicationELB"
      period      = 120
      stat        = "Sum"
      unit        = "Count"

      dimensions = {
        LoadBalancer = data.aws_lb.ingress.arn_suffix
      }
    }
  }

  metric_query {
    id = "m2"

    metric {
      metric_name = "HTTPCode_ELB_4XX_Count"
      namespace   = "AWS/ApplicationELB"
      period      = 120
      stat        = "Sum"
      unit        = "Count"

      dimensions = {
        LoadBalancer = data.aws_lb.ingress.arn_suffix
      }
    }
  }
}

this is the example alarm on the terraform page https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_metric_alarm.
The apply step works fine when the ticket is merged, but the subsequent PRs that we merge fail on the apply step with the same error, blocking the pipeline