ALBs need v2 WAF (ODC EKS module)

[benjimin]

The ODC EKS module creates a v1 WAF, which is obsolete (deprecated by AWS):

datacube-k8s-eks/odc_eks/waf.tf

Line 458 in efeb5a3

resource "aws_wafregional_web_acl" "waf_webacl" {

We need the cluster to use a (non-deprecated) v2 WAF.

Moreover, ensuring the v2 WAF stays in front of the k8s services (when k8s conventionally creates ALBs dynamically) is not straight forward. Some possible options are:

• Terraform an ALB for the cluster (complete with WAF). Also, can terraform Route53 wildcard record to point at the ALB, obviating any need for external-dns controller. (Let the ALB serve a 404 for nonexistent subdomains.) Tag the ALB so that the ALB controller (in k8s) will recognise it and try to manage the rules and target groups therein. Update the IAM roles (associated with the ALB controller's service account) to deny it permission to destroy the ALB or create other ALBs. If necessary, annotate k8s ingresses to share the same ingress group (corresponding to the terraformed ALB).
• Let the ALB controller create the WAF. (This may reduce control over the WAF settings.)
• Terraform the WAF but let the ALB controller attach it. This can be done by annotating one of the ingresses (from the ingress-group) with the id of the WAF. (If somehow detached, the controller typically will try to reattach every 10hrs.)
• Terraform AWS Firewall Manager to automatically attach the desired WAF whenever k8s creates an ALB. (I think this service costs about $1kpa.)

I like the first option. I think it simplifies the configuration (and information flow) overall, and it lets terraform de-provision the entire cluster without leaving expensive load balancers orphaned. (Especially if node instances are contained in terraformed ASGs, as when managing by cluster-autoscaler rather than karpenter.) However, it's currently an off-label usage of the controller. (But avoids needing to self-manage additional k8s resources.)

Even if this module doesn't attach the v2 WAF, at least creating the v2 WAF should ensure the logging is configured appropriately (which would be tricky to set up by clickops for more than one cluster).

[benjimin]

@ankurbhardwa has proposed another possible option, including for completeness:

• Have terraform recognise the cluster's ALB(s) via tags and attach the WAF, and manually rerun the terraform pipeline after any occasion that the ALB controller (re)creates an ALB.

[ankurbhardwa]

i have used custom script and implement this way.

Finds all ALBs in account/region

Filters only Application Load Balancers (Type == "application")

Filters only those with the tag elbv2.k8s.aws/cluster = cluster_id

Associates WAFv2 ACL