panic: checkable object status report for unexpected checkable object after 1.13.3 upgrade

[faust64]

Terraform Version

Terraform v1.13.3
on linux_amd64

Terraform Configuration Files

module "my_platform_coordinates" {
  for_each       = toset(contains(var.features_opt-out, "chart-remote-monitoring-stack") ? [] : [var.platform.stage])
  platform_stage = each.key
  source         = "tfe.corp.com/Global-Module-Sharing/cmdb/corp"
}

Debug Output

panic: checkable object status report for unexpected checkable object module.kube_components.module.my_platform_coordinates["dev"].var.platform_stage
goroutine 11579 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:26 +0x5e
github.com/hashicorp/terraform/internal/logging.PanicHandler()
	github.com/hashicorp/terraform/internal/logging/panic.go:84 +0x16a
panic({0x34fe320?, 0xc01b7d7400?})
	runtime/panic.go:792 +0x132
github.com/hashicorp/terraform/internal/terraform.(*Graph).walk.func1.1()
	github.com/hashicorp/terraform/internal/terraform/graph.go:59 +0x4c5
panic({0x34fe320?, 0xc01b7d7400?})
	runtime/panic.go:792 +0x132
github.com/hashicorp/terraform/internal/checks.(*State).reportCheckResult(0xc009272cd8, {0x44a9ce0, 0xc00dc86990}, 0x6, 0x0, 0x50)
	github.com/hashicorp/terraform/internal/checks/state_report.go:108 +0x345
github.com/hashicorp/terraform/internal/checks.(*State).ReportCheckResult(0x3aa6f20?, {0x44a9ce0?, 0xc00dc86990?}, 0xc00e009360?, 0x410?, 0xf?)
	github.com/hashicorp/terraform/internal/checks/state_report.go:73 +0xc5
github.com/hashicorp/terraform/internal/terraform.evalVariableValidations({{0xc00dc84540, 0x2, 0x2}, {{}, {0xc004905d00, 0xe}}}, {0x44cd0c0, 0xc018c1ae00}, {0xc0069c16d0, 0x1, ...}, ...)
	github.com/hashicorp/terraform/internal/terraform/eval_variable.go:322 +0x1072
github.com/hashicorp/terraform/internal/terraform.(*nodeVariableValidation).Execute(0xc014ecb680, {0x44cd0c0, 0xc014715300}, 0x50?)
	github.com/hashicorp/terraform/internal/terraform/node_variable_validation.go:126 +0x23e
github.com/hashicorp/terraform/internal/terraform.(*ContextGraphWalker).Execute(0xc012ba94a0, {0x44cd0c0, 0xc014715300}, {0x7fbe708b2080, 0xc014ecb680})
	github.com/hashicorp/terraform/internal/terraform/graph_walk_context.go:163 +0xad
github.com/hashicorp/terraform/internal/terraform.(*Graph).walk.func1({0x399fd00, 0xc014ecb680})
	github.com/hashicorp/terraform/internal/terraform/graph.go:143 +0x7c3
github.com/hashicorp/terraform/internal/dag.(*Walker).walkVertex(0xc007ff8540, {0x399fd00, 0xc014ecb680}, 0xc005e88d80)
	github.com/hashicorp/terraform/internal/dag/walk.go:393 +0x2d1
created by github.com/hashicorp/terraform/internal/dag.(*Walker).Update in goroutine 356
	github.com/hashicorp/terraform/internal/dag/walk.go:316 +0xf33

Expected Behavior

terraform shouldn't have crashed.
should have released its lock

Actual Behavior

terraform crashed

Steps to Reproduce

unclear, saw that issue once

Additional Context

I upgraded our dev envs from terraform 1.12.2 to 1.13.3 couple days ago.
Yesterday, colleague of mine brought this up to me.

Weirdly enough, the apply that failed, ran from a chain, where that module nor its parent had no diffs. Chain should have changed one local file, and some kubernetes object, completely unrelated, no shared variables, no dependencies

TFE module involved had no changes in the last 2 weeks. Returns with a list of environments metadata, from local yaml files/part of that TFE module, based on search filters from inputs/variables.
In that TFE module, data is loaded with:

locals {
  matched_platforms = fileset("${path.module}/data/", "*.yml")
  known_platforms = {
    for i in local.matched_platforms :
    replace(reverse(split("/", i))[0], ".yml", "") => yamldecode(file("${path.module}/data/${i}"))
  }
...

Then input filters are applied, only to return matches.

Chain was generated few minutes before running apply.
Conditions to include that module or not did not change -- in this case, it shouldn't be included at all, ternary should resolve to []. Can't find any trace of that module in plan, beyond it being cloned at terraform init time.

Still we should note: if the condition checked in that ternary returned otherwise, that my_platform_coordinates["dev"] is exactly what I would have expected to find in my terraform plan logs.

I couldn't find any changes in git either, that may suggest ternary's conditions would have resulted in inclusion.
Object wasn't in chain to be applied ...

Can't make sense out of it.
Seen only once. That pipeline has been running fine before and after. Other dev environments are fine as well.
Still, from the traces and recent changelogs, I'm a bit suspicious about https://github.com/hashicorp/terraform/pull/37001/files

Should I be worried about promoting 1.13.3 further?
Any chance this would be unrelated to upgrading terraform?
What could it be?

Sorry I don't have much more. Let me know if there's anything else I can test trying to reproduce. Or if I'm just over-reacting.

Thanks.

References

No response

Generative AI / LLM assisted development?

No response

[jbardin]

Hi @faust64,

Thanks for filing the issue! Definitely not an overreaction, anytime you see a panic it represents a bug that we should fix, rather than some sort of transient runtime error. I doubt this is directly related to the upgrade, and it's definitely unrelated to the linked PR which has no impact on variable validations or graph structure. We've had a few reports with a very similar stack trace before, but none were followed up on with enough information to debug it, so I don't think it's something unique to 1.13, it's just a difficult to reproduce.

If you can produce a trace log with TF_LOG_CORE=trace, even with a successful apply, that might help track down the issue. It's going to be something related to the particular structure of the configuration and variable validation, so any related configuration would help too.

Thanks!