feat: Add Route53 entry and IAM role for GitHub Actions OIDC integration (#12)

* feat: add Route53 entry and IAM role for GH Actions OIDC, update mise configuration

* fix: correct syntax for token.actions.githubusercontent.com in IAM role policy

* Update cloudformation/route53-gh-action-iam-role-oidc.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: correct casing for ManagedPolicyArns in IAM role CloudFormation template

* fix: correct casing for ManagedPolicyArns in CloudFormation deployment script

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Author: ruzickap

.gitignore

@@ -1,2 +1,6 @@
 # pre-commit configuration file
 .pre-commit-config.yaml
+
+# mise local files (https://mise.jdx.dev/configuration/environments.html#config-environments)
+mise.local.toml
+mise.*.local.toml

README.md

@@ -1,23 +1,130 @@
 # k8s-multicluster-gitops
 
-Infrastructure as Code for provisioning multiple Kubernetes clusters, managed
-using GitOps with ArgoCD
+Infrastructure as Code (IaC) for provisioning and managing multiple Kubernetes
+clusters across multiple cloud accounts, using GitOps principles with ArgoCD.
 
-Create all "kind" clusters:
+## Requirements
+
+Guides on setting up Kubernetes clusters in the cloud are common, but few cover
+managing clusters across multiple providers and accounts (a key need for large
+enterprises)
+
+This project aims to provide a practical example:
+
+- ✅ Provisioning and managing Kubernetes clusters across multiple cloud
+  providers (AWS, Azure, GCP).
+- ✅ Deploying and maintaining Kubernetes clusters across multiple accounts and
+  regions.
+
+## Architecture
+
+You likely need to deploy multiple Kubernetes clusters across various cloud
+providers and accounts.
+
+Each cloud provider has a designated "primary account" where subdomains are hosted:
+
+- `aws.mylabs.dev` - AWS
+- `az.mylabs.dev` - Azure
+- `gcp.mylabs.dev` - Google Cloud Platform
+
+> The second-level domain `mylabs.dev` is hosted externally (e.g., Cloudflare),
+> and it's the user's responsibility to configure DNS delegation properly.
+
+An IAM role (or the equivalent for each cloud provider) will be created in the
+primary account. This role will allow GitHub Actions / mise to manage resources
+in the primary account and will also be used to access other accounts where
+Kubernetes clusters are deployed.
+
+## Cloud Providers - Multi-Account Setup
+
+Let's assume you have 2 AWS accounts, 2 Azure accounts, 2 GCP accounts and you
+want to deploy 2 Kubernetes clusters (EKS, AKS, GKE) in each account:
+
+| Cloud Provider                                   | Account 01                                                   | Account 02                                                   |
+|--------------------------------------------------|--------------------------------------------------------------|--------------------------------------------------------------|
+| **AWS** (`aws.mylabs.dev`, `k8s.aws.mylabs.dev`) | `k01.k8s.aws.mylabs.dev` (US), `k02.k8s.aws.mylabs.dev` (EU) | `k03.k8s.aws.mylabs.dev` (US), `k04.k8s.aws.mylabs.dev` (EU) |
+| **Azure** (`az.mylabs.dev`, `k8s.az.mylabs.dev`) | `k01.k8s.az.mylabs.dev` (US), `k02.k8s.az.mylabs.dev` (EU)   | `k03.k8s.az.mylabs.dev` (US), `k04.k8s.az.mylabs.dev` (EU)   |
+| **GCP** (`gcp.mylabs.dev`, `k8s.gcp.mylabs.dev`) | `k01.k8s.gcp.mylabs.dev` (US), `k02.k8s.gcp.mylabs.dev` (EU) | `k03.k8s.gcp.mylabs.dev` (US), `k04.k8s.gcp.mylabs.dev` (EU) |
+
+### AWS
+
+#### Primary account
+
+Choose one of your AWS accounts to act as the **primary account** and create a
+Route 53 hosted zone for `aws.mylabs.dev`
+
+> Ensure that the necessary environment variables are set for the AWS CLI
+> (e.g., `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`).
+
+```bash
+mise run create:aws-primary:cf-route53-gh-action-iam-role-oidc
+```
+
+> For more details please inspect the [mise.toml](./mise.toml) file.
+
+Next, manually set up the DNS delegation between your second-level domain
+`mylabs.dev` and the `aws.mylabs.dev` hosted zone in Route 53.
+
+Example:
+
+![Cloudflare DNS records for mylabs.dev](images/cloudflare-mylabs-dev-dns-records.avif)
+
+#### Tenant Accounts
+
+Create an IAM role in each tenant account that allows the primary account to
+assume a role in the tenant account.
+
+> Make sure to use AWS credentials (`AWS_ACCESS_KEY_ID`,
+> `AWS_SECRET_ACCESS_KEY`) for the tenant account.
+
+```bash
+mise run create:aws-tenant:cf-iam-role
+```
+
+### Azure
+
+### GCP
+
+## K8s Clusters
+
+All the "kubeconfig files" will be stored in the `clusters/.kubeconfigs`
+directory.
+
+### Kind
+
+The [kind clusters](https://kind.sigs.k8s.io/) are created using the `kind`
+tool, which is a tool for running Kubernetes clusters in Docker containers.
 
 ```bash
-mise task run "create:kind:*"
-mise task run "delete:kind:*"
+mise run create:kind:kind01-internal
+mise run delete:kind:kind01-internal
+mise run create:kind:kind02-internal
+mise run delete:kind:kind02-internal
+mise run create-kind-all
+mise run delete-kind-all
 ```
 
-Create all "k3d" clusters:
+### K3d
+
+The [k3d clusters](https://k3d.io/) are created using the `k3d` tool, which
+is a lightweight wrapper to run `k3s` in Docker containers.
 
 ```bash
-mise task run "create:k3d:*"
-mise task run "delete:k3d:*"
+mise run create:k3d:k3d01-internal
+mise run delete:k3d:k3d01-internal
+mise run create:k3d:k3d02-internal
+mise run delete:k3d:k3d02-internal
+mise run create-k3d-all
+mise run delete-k3d-all
 ```
 
-> Same for eksctl, az, terraform-aws, terraform-az, ... clusters
+> You can also create all the clusters at once using the `create-all` and
+> `delete-all` commands:
+>
+> ```bash
+> mise run "create-kind-all" ::: "create-k3d-all"
+> mise run "delete-kind-all" ::: "delete-k3d-all"
+> ```
 
 ## Architecture diagrams
 
@@ -34,10 +141,6 @@ flowchart TB
       aws.mylabs.dev@{ icon: "logos:aws-route53", form: "circle", label: "aws.mylabs.dev", pos: "b", h: 60 }
       k8s.aws.mylabs.dev@{ icon: "logos:aws-route53", form: "square", label: "k8s.aws.mylabs.dev", pos: "b", h: 60 }
     end
-    subgraph "AWS Account 03"
-      k05.k8s.aws.mylabs.dev@{ icon: "logos:aws-route53", form: "square", label: "k05.k8s.aws.mylabs.dev", pos: "b", h: 60 }
-      k06.k8s.aws.mylabs.dev@{ icon: "logos:aws-route53", form: "square", label: "k06.k8s.aws.mylabs.dev", pos: "b", h: 60 }
-    end
     subgraph "AWS Account 02"
       k03.k8s.aws.mylabs.dev@{ icon: "logos:aws-route53", form: "square", label: "k03.k8s.aws.mylabs.dev", pos: "b", h: 60 }
       k04.k8s.aws.mylabs.dev@{ icon: "logos:aws-route53", form: "square", label: "k04.k8s.aws.mylabs.dev", pos: "b", h: 60 }
@@ -53,10 +156,6 @@ flowchart TB
       az.mylabs.dev@{ icon: "logos:azure-icon", form: "circle", label: "az.mylabs.dev", pos: "b", h: 60 }
       k8s.az.mylabs.dev@{ icon: "logos:azure-icon", form: "square", label: "k8s.az.mylabs.dev", pos: "b", h: 60 }
     end
-    subgraph "Azure Account 03"
-      k05.k8s.az.mylabs.dev@{ icon: "logos:azure-icon", form: "square", label: "k05.k8s.az.mylabs.dev", pos: "b", h: 60 }
-      k06.k8s.az.mylabs.dev@{ icon: "logos:azure-icon", form: "square", label: "k06.k8s.az.mylabs.dev", pos: "b", h: 60 }
-    end
     subgraph "Azure Account 02"
       k03.k8s.az.mylabs.dev@{ icon: "logos:azure-icon", form: "square", label: "k03.k8s.az.mylabs.dev", pos: "b", h: 60 }
       k04.k8s.az.mylabs.dev@{ icon: "logos:azure-icon", form: "square", label: "k04.k8s.az.mylabs.dev", pos: "b", h: 60 }
@@ -72,10 +171,6 @@ flowchart TB
       gcp.mylabs.dev@{ icon: "logos:google-cloud", form: "circle", label: "gcp.mylabs.dev", pos: "b", h: 60 }
       k8s.gcp.mylabs.dev@{ icon: "logos:google-cloud", form: "square", label: "k8s.gcp.mylabs.dev", pos: "b", h: 60 }
     end
-    subgraph "GCP Account 03"
-      k05.k8s.gcp.mylabs.dev@{ icon: "logos:google-cloud", form: "square", label: "k05.k8s.gcp.mylabs.dev", pos: "b", h: 60 }
-      k06.k8s.gcp.mylabs.dev@{ icon: "logos:google-cloud", form: "square", label: "k06.k8s.gcp.mylabs.dev", pos: "b", h: 60 }
-    end
     subgraph "GCP Account 02"
       k03.k8s.gcp.mylabs.dev@{ icon: "logos:google-cloud", form: "square", label: "k03.k8s.gcp.mylabs.dev", pos: "b", h: 60 }
       k04.k8s.gcp.mylabs.dev@{ icon: "logos:google-cloud", form: "square", label: "k04.k8s.gcp.mylabs.dev", pos: "b", h: 60 }
@@ -92,28 +187,20 @@ flowchart TB
   k8s.aws.mylabs.dev --> k02.k8s.aws.mylabs.dev
   k8s.aws.mylabs.dev --> k03.k8s.aws.mylabs.dev
   k8s.aws.mylabs.dev --> k04.k8s.aws.mylabs.dev
-  k8s.aws.mylabs.dev --> k05.k8s.aws.mylabs.dev
-  k8s.aws.mylabs.dev --> k06.k8s.aws.mylabs.dev
   mylabs.dev --> az.mylabs.dev
   az.mylabs.dev --> k8s.az.mylabs.dev
   k8s.az.mylabs.dev --> k01.k8s.az.mylabs.dev
   k8s.az.mylabs.dev --> k02.k8s.az.mylabs.dev
   k8s.az.mylabs.dev --> k03.k8s.az.mylabs.dev
   k8s.az.mylabs.dev --> k04.k8s.az.mylabs.dev
-  k8s.az.mylabs.dev --> k05.k8s.az.mylabs.dev
-  k8s.az.mylabs.dev --> k06.k8s.az.mylabs.dev
   mylabs.dev --> gcp.mylabs.dev
   gcp.mylabs.dev --> k8s.gcp.mylabs.dev
   k8s.gcp.mylabs.dev --> k01.k8s.gcp.mylabs.dev
   k8s.gcp.mylabs.dev --> k02.k8s.gcp.mylabs.dev
   k8s.gcp.mylabs.dev --> k03.k8s.gcp.mylabs.dev
   k8s.gcp.mylabs.dev --> k04.k8s.gcp.mylabs.dev
-  k8s.gcp.mylabs.dev --> k05.k8s.gcp.mylabs.dev
-  k8s.gcp.mylabs.dev --> k06.k8s.gcp.mylabs.dev
 ```
 
----
-
 ## Tests
 
 ```bash

cloudformation/aws-cf-route53-gh-action-iam-role-oidc.yml

@@ -0,0 +1,88 @@
+AWSTemplateFormatVersion: 2010-09-09
+
+Description: Route53 entry and IAM role for GitHub Actions OIDC federated role
+
+Parameters:
+  PrimaryDomain:
+    Description: "Primary domain for AWS services (e.g. aws.mylabs.dev)"
+    Type: String
+  K8sDomain:
+    Description: "K8s domain hosting cluster subdomains (e.g. k8s.aws.mylabs.dev)"
+    Type: String
+  GithubActionsThumbprint:
+    Type: CommaDelimitedList
+    Default: 6938fd4d98bab03faadb97b34396831e3780aea1
+    Description: >
+      Comma separated list of thumbprints for GitHub Actions tokens.
+      Default comes from https://github.blog/changelog/2022-01-13-github-actions-update-on-oidc-based-deployments-to-aws/
+  AudienceList:
+    Type: CommaDelimitedList
+    Default: sts.amazonaws.com
+    Description: >
+      Comma separated list of allowed audience for the tokens.
+      Default is audience for the official AWS configure action from https://github.com/aws-actions/configure-aws-credentials
+  SubjectClaimFilters:
+    Type: CommaDelimitedList
+    Description: >
+      Subject claim filter for valid tokens.
+      Default allows any branch or tag of the McK-Internal/VM-wiz-automation to assume the role.
+      See https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#example-subject-claims
+      for examples of filtering by branch or deployment environment.
+  ManagedPolicyArns:
+    Type: CommaDelimitedList
+    Description: Comma separated list for arns for managed policies to attach to the role
+  RoleName:
+    Description: "Name of the new IAM role used by GitHub Actions"
+    Type: String
+
+Resources:
+  PrimaryHostedZone:
+    Type: AWS::Route53::HostedZone
+    Properties:
+      Name: !Ref PrimaryDomain
+      HostedZoneConfig:
+        Comment: Primary domain for AWS services
+  K8sHostedZone:
+    Type: AWS::Route53::HostedZone
+    Properties:
+      Name: !Ref K8sDomain
+      HostedZoneConfig:
+        Comment: K8s domain hosting cluster subdomains
+  nsRootPrimaryHostedZoneRecordSet:
+    Type: "AWS::Route53::RecordSet"
+    Properties:
+      HostedZoneId: !Ref PrimaryHostedZone
+      Name: !Sub "${K8sDomain}."
+      Type: NS
+      TTL: 86400
+      ResourceRecords: !GetAtt K8sHostedZone.NameServers
+  GitHubIdentityProvider:
+    Type: AWS::IAM::OIDCProvider
+    Properties:
+      ClientIdList: !Ref AudienceList
+      ThumbprintList: !Ref GithubActionsThumbprint
+      Url: https://token.actions.githubusercontent.com
+  GitHubActionsServiceRole:
+    Type: AWS::IAM::Role
+    Properties:
+      AssumeRolePolicyDocument:
+        Version: "2012-10-17"
+        Statement:
+          - Sid: RoleForGitHubActions
+            Effect: Allow
+            Principal:
+              Federated: !Ref GitHubIdentityProvider
+            Action:
+              - "sts:AssumeRoleWithWebIdentity"
+            Condition:
+              StringLike:
+                "token.actions.githubusercontent.com:sub": !Ref SubjectClaimFilters
+      Description: Service Role for use in GitHub Actions
+      RoleName: !Ref RoleName
+      MaxSessionDuration: 36000
+      ManagedPolicyArns: !Ref ManagedPolicyArns
+
+Outputs:
+  GitHubActionsServiceRoleArn:
+    Description: Arn of service role for use in GitHub actions
+    Value: !GetAtt GitHubActionsServiceRole.Arn