Warning on Duplicate keys in a map object.

[techdragon]

Current Terraform Version

Terraform v0.15.0
on darwin_amd64

Use-cases

Avoiding duplicate keys in maps.

Attempted Solutions

Linter scripts and third party tools.

Proposal

Duplicate keys in maps should raise some kind of warning. This would help prevent mistakes when working with larger and more complex maps. In particular dynamic maps and merged maps where the keys may not be visible in the terraform configuration as written, only being populated when terraform is run.

References

N.A.

[myronmeier]

Thanks for raising this, @techdragon, literally days before we ran into the same issue. Upvoted! I would even go so far as to propose it be an error and not just a warning.

Here is a simple demonstration.

terraform {
  required_version = "0.15.4"
}

locals {
  local_dbs = {
    db1 = { a = "1a" }
    db1 = { a = "2a" }    // notice the key here is the same as the previous (e.g. a mistake)
    db3 = { a = "3a" }
  }
}

output locals_map_out {
  value = local.local_dbs
}

The above produces these outputs

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

locals_map_out = {
  "db1" = {
    "a" = "2a"
  }
  "db3" = {
    "a" = "3a"
  }
}

Notice the loss of the 1st db1 map entry. There was no error and no warning. It just silently replaced the 1st map entry with the 2nd.

The same can be demonstrated with input variables as well

variable "var_dbs" {
  type = map(object({
    a = string
  }))
}

output variable_map_out {
  value = var.var_dbs
}

With this terraform.tfvars

var_dbs = {
  db1 = { a = "1a" }
  db1 = { a = "2a" }
  db3 = { a = "3a" }
}

Produces

Outputs:

variable_map_out = tomap({
  "db1" = {
    "a" = "2a"
  }
  "db3" = {
    "a" = "3a"
  }
})

Adding more outputs to the locals example above to see if the 1st map entry is just somehow hidden indicates that it was likely lost while constructing the map itself. This similar older issue #18573 (circa TF 11?) seems to confirm that, but it also makes a prediction about duplicate map keys being an error in future versions (TF 12?), which appears to have not come true.

output locals_map_length {
  value = length(local.local_dbs)
}

output locals_map_keys {
  value = keys(local.local_dbs)
}

output locals_map_values {
  value = values(local.local_dbs)
}

output locals_map_json {
  value = jsonencode(local.local_dbs)
}

Outputs:

locals_map_json = "{\"db1\":{\"a\":\"2a\"},\"db3\":{\"a\":\"3a\"}}"
locals_map_keys = [
  "db1",
  "db3",
]
locals_map_length = 2
locals_map_out = {
  "db1" = {
    "a" = "2a"
  }
  "db3" = {
    "a" = "3a"
  }
}
locals_map_values = [
  {
    "a" = "2a"
  },
  {
    "a" = "3a"
  },
]

Even reading from JSON silently drops the duplicate key without error or warning

locals {
  local_json = jsondecode( <<-EOT
    {
      "db1": { "a": "1a" },
      "db1": { "a": "2a" },
      "db3": { "a": "3a" }
    }
    EOT
  )
}

output locals_json_out {
  value = local.local_json
}

Outputs:

locals_json_out = {
  "db1" = {
    "a" = "2a"
  }
  "db3" = {
    "a" = "3a"
  }
}

[myronmeier]

@techdragon , I'm curious what linters you tried. I quickly tried https://github.com/terraform-linters/tflint but it didn't catch the duplicate keys in any of the examples above.

[techdragon]

@myronmeier Thanks for writing out the excellent examples! And yes I had the same issue tflint not catching it, but since the issue goes beyond the maps as written statically in code, I just mentioned the linters in passing as I had tried them as a workaround, but I probably should have been more clear about what I meant.

[nwsparks]

@myronmeier I agree, this should be an error. Warning is not sufficient as it's easily missed and this can be very problematic.

From what I can tell this can't be caught by variable validation either.

nsparks@w10-vm:/mnt/c/test$ cat data.tf
variable "routing_rules" {
  default =   {
    "8000" = {
      host_headers = ["h1"]
    },
    "8000" = {
      host_headers = ["h2"]
    },
    "8001" = {
      host_headers = ["h3"]
    }
  }

  validation {
    condition = length(keys(var.routing_rules)) == 3
    error_message = "Length is not 3."
  }
}
nsparks@w10-vm:/mnt/c/test$ terraform console
╷
│ Warning: Due to the problems above, some expressions may produce unexpected results.
│
│
╵

╷
│ Error: Invalid value for variable
│
│   on data.tf line 1:
│    1: variable "routing_rules" {
│
│ Length is not 3.
│
│ This was checked by the validation rule at data.tf:14,3-13.
╵

[techdragon]

@nwsparks I didn't expect validation to fail like that... Also I'm not 100% convinced that it should always be an error. I can imagine there are users out there who may be relying on this to power various workarounds for providing default variable values that may have been inherited from older terraform modules or when needing to pass in configuration and merge defaults with more complex map related values. While it might be good to make this an error in time, I'd rather have the warning added now, with perhaps some way to configure this warning as an error, than navigate the longer path of changing behaviour like this now that 1.0 released and the terraform team have more to worry about with respect to long term compatibility.

Edit: Just found another example of duplicated key names that may become a common pattern in future that could conflict with making duplicate keys an error #25088 has an example where the key is defined multiple times for running different validation checks against its value.

[philip-harvey]

I struck this today, and it's very dangerous. In terraform.tfvars it will happily let you define a map with the same key multiple times as per the example above and doesn't throw an error, but it discards all but one of the map objects with that key. This can lead to non-recoverable data loss. I tried to add a test to check for duplicate keys, but it seems to discard the data right when it reads it in so there doesn't seems to be any way to test for this inside Terraform. I suggest that Terraform should error if you have multiple occurrences of the same key in a map in terraform.tfvars.

[fernando-villalba]

This one is really bizarre that you can have duplicate keys for objects and maps in terraform as it can be a considerable to debug headache and potentially catastrophic if not careful. Neither Python, Java or Golang accept duplicate keys for maps so TF accepting them is just strange

What's even stranger is how very few downvotes this has! I hope it gets addressed soon.

[ams1]

For anyone interested in workarounds for this, I opened a SO question:
https://stackoverflow.com/questions/74233973/terraform-map-alternative-when-duplicated-keys-should-raise-error/74234215#74234215

[naitmare01]

I managed to solve it by first creating a list of items containing my key-value pair in my map. Then I converted it to a map, the conversion itself will check for duplicates and error out if any exist.

locals{
tuple_items = [
{ "AB" = {
ID = "123"
} },
{ "CD" = {
ID = "456"
} }
]
tuple_to_map = { for item in local.tuple_items : keys(item)[0] => values(item)[0] } # Transform the tuple to a map to be able to iterate over it, output errors if not all keys are unique
}