Skip to content

start managing azure infra with terraform#8974

Open
upodroid wants to merge 1 commit into
kubernetes:mainfrom
upodroid:az-terraform
Open

start managing azure infra with terraform#8974
upodroid wants to merge 1 commit into
kubernetes:mainfrom
upodroid:az-terraform

Conversation

@upodroid
Copy link
Copy Markdown
Member

This PR introduces a few changes:

  1. Adds a new Terraform layer called root that handles entra id and privileged resources such as subscriptions, management groups
  2. Allows Atlantis to access Azure via WI
  3. All prow CI runs in the same sub where our build cluster runs. This PR creates the scaffolding to move CI to its own subscription and allows it to run directly from our build cluster.

@k8s-ci-robot k8s-ci-robot added area/infra Infrastructure management, infrastructure design, code in infra/ area/infra/azure Issues or PRs related to Kubernetes Azure infrastructure area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. labels Jan 20, 2026
@k8s-ci-robot k8s-ci-robot requested review from aojea and jbpratt January 20, 2026 21:04
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Jan 20, 2026
@k8s-infra-ci-robot

This comment has been minimized.

Comment thread infra/azure/terraform/README.md Outdated
@k8s-infra-ci-robot

This comment has been minimized.

@k8s-infra-ci-robot
Copy link
Copy Markdown
Contributor

k8s-infra-ci-robot commented Jan 20, 2026

Argo CD Diff Preview

Summary:

Total: 2 files changed

Modified (2):
± aks-prow-build (+16)
± atlantis (+21)
aks-prow-build (kubernetes/apps/prow.yaml)
@@ Application modified: aks-prow-build (kubernetes/apps/prow.yaml) @@
       }
     }
 kind: ConfigMap
 metadata:
   name: google-adc
   namespace: test-pods
 ---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  annotations:
+    azure.workload.identity/client-id: 333bb18b-207b-4abd-9ed0-e7e3834378b1
+  name: azure
+  namespace: test-pods
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  annotations:
+    azure.workload.identity/client-id: f23f8fcc-855b-40fd-a41b-b329ccdb95a1
+  name: rg-cleanup
+  namespace: test-pods
+---
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
   annotations:
     api-approved.kubernetes.io: https://github.com/kubernetes-sigs/boskos/pull/105
   name: dynamicresourcelifecycles.boskos.k8s.io
 spec:
atlantis (kubernetes/apps/atlantis.yaml)
@@ Application modified: atlantis (kubernetes/apps/atlantis.yaml) @@
       - env:
         - name: AWS_ROLE_ARN
           value: arn:aws:iam::348685125169:role/atlantis
         - name: AWS_WEB_IDENTITY_TOKEN_FILE
           value: /var/run/secrets/aws-iam-token/serviceaccount/token
         - name: AWS_REGION
           value: us-east-2
+        - name: ARM_USE_AKS_WORKLOAD_IDENTITY
+          value: "true"
+        - name: ARM_SUBSCRIPTION_ID
+          value: 46678f10-4bbb-447e-98e8-d2829589f2d8
+        - name: AZURE_CLIENT_ID
+          value: 6fe87cee-6470-45d8-accc-57687193e504
+        - name: AZURE_FEDERATED_TOKEN_FILE
+          value: /var/run/secrets/azure-token/serviceaccount/token
+        - name: AZURE_TENANT_ID
+          value: d1aa7522-0959-442e-80ee-8c4f7fb4c184
         - name: ATLANTIS_CONFIG
           value: /config/atlantis.yaml
         - name: ATLANTIS_GH_TOKEN
           valueFrom:
             secretKeyRef:
               key: token
               name: atlantis-vcs
@@ skipped 32 lines (41 -> 72) @@
             memory: 1Gi
         volumeMounts:
         - mountPath: /config
           name: config
         - mountPath: /var/run/secrets/aws-iam-token/serviceaccount
           name: aws-iam-token
           readOnly: true
+        - mountPath: /var/run/secrets/azure-token/serviceaccount
+          name: azure-token
+          readOnly: true
         - mountPath: /atlantis
           name: atlantis-data
       securityContext:
         fsGroup: 1000
       serviceAccountName: atlantis
       volumes:
       - configMap:
           name: atlantis-config-4mc949mdm2
         name: config
       - name: aws-iam-token
         projected:
           defaultMode: 420
           sources:
           - serviceAccountToken:
               audience: sts.amazonaws.com
+              expirationSeconds: 86400
+              path: token
+      - name: azure-token
+        projected:
+          defaultMode: 420
+          sources:
+          - serviceAccountToken:
+              audience: api://AzureADTokenExchange
               expirationSeconds: 86400
               path: token
   updateStrategy:
     rollingUpdate:
       partition: 0
     type: RollingUpdate
   volumeClaimTemplates:

Stats:
[Applications: 74], [Full Run: 1m34s], [Rendering: 1m29s], [Cluster: 0s], [Argo CD: 1s]

@upodroid
Copy link
Copy Markdown
Member Author

atlantis plan

@k8s-infra-ci-robot

This comment has been minimized.

@upodroid
Copy link
Copy Markdown
Member Author

atlantis plan

@k8s-infra-ci-robot

This comment has been minimized.

@upodroid
Copy link
Copy Markdown
Member Author

atlantis plan

@upodroid upodroid removed the request for review from aojea January 20, 2026 22:15
@k8s-infra-ci-robot

This comment has been minimized.

@upodroid
Copy link
Copy Markdown
Member Author

atlantis plan

@k8s-infra-ci-robot

This comment has been minimized.

@upodroid
Copy link
Copy Markdown
Member Author

atlantis plan

@k8s-infra-ci-robot
Copy link
Copy Markdown
Contributor

Ran Plan for dir: infra/azure/terraform/root workspace: default

Plan Error

Show Output
running 'sh -c' '/usr/local/bin/terraform plan -input=false -refresh -out "/atlantis/repos/kubernetes/k8s.io/8974/default/infra/azure/terraform/root/default.tfplan"' in '/atlantis/repos/kubernetes/k8s.io/8974/default/infra/azure/terraform/root': exit status 1
Acquiring state lock. This may take a few moments...
module.role_assignments.data.modtm_module_source.telemetry[0]: Reading...
module.role_assignments.random_uuid.telemetry[0]: Refreshing state... [id=016e026a-dbc2-ff65-38c6-8554c6161445]
module.role_assignments.data.modtm_module_source.telemetry[0]: Read complete after 0s
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["owner"]: Reading...
module.role_assignments.data.azurerm_client_config.telemetry[0]: Reading...
module.role_assignments.data.azurerm_client_config.current: Reading...
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["contributor"]: Reading...
module.role_assignments.data.azurerm_management_group.management_groups_by_id_or_display_name["root"]: Reading...
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["monitoring-reader"]: Reading...
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["reader"]: Reading...
module.role_assignments.data.azurerm_client_config.current: Read complete after 0s [id=Y2xpZW50Q29uZmlncy9jbGllbnRJZD02ZmU4N2NlZS02NDcwLTQ1ZDgtYWNjYy01NzY4NzE5M2U1MDQ7b2JqZWN0SWQ9NDE4MDk0MTYtZTI4MC00OGUxLTgzNTQtODQ4M2FmMTkwZmJjO3N1YnNjcmlwdGlvbklkPTQ2Njc4ZjEwLTRiYmItNDQ3ZS05OGU4LWQyODI5NTg5ZjJkODt0ZW5hbnRJZD1kMWFhNzUyMi0wOTU5LTQ0MmUtODBlZS04YzRmN2ZiNGMxODQ=]
module.role_assignments.data.azurerm_client_config.telemetry[0]: Read complete after 0s [id=Y2xpZW50Q29uZmlncy9jbGllbnRJZD02ZmU4N2NlZS02NDcwLTQ1ZDgtYWNjYy01NzY4NzE5M2U1MDQ7b2JqZWN0SWQ9NDE4MDk0MTYtZTI4MC00OGUxLTgzNTQtODQ4M2FmMTkwZmJjO3N1YnNjcmlwdGlvbklkPTQ2Njc4ZjEwLTRiYmItNDQ3ZS05OGU4LWQyODI5NTg5ZjJkODt0ZW5hbnRJZD1kMWFhNzUyMi0wOTU5LTQ0MmUtODBlZS04YzRmN2ZiNGMxODQ=]
module.role_assignments.modtm_telemetry.telemetry[0]: Refreshing state... [id=af03c3da-3051-4ba2-b39b-f9d89929dbf6]
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["reader"]: Read complete after 0s [id=/providers/Microsoft.Authorization/roleDefinitions/acdd72a7-3385-48ef-bd42-f606fba81ae7]
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["contributor"]: Read complete after 0s [id=/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c]
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["owner"]: Read complete after 0s [id=/providers/Microsoft.Authorization/roleDefinitions/8e3af657-a8ff-443c-a75c-2fe8c4bcb635]
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["monitoring-reader"]: Read complete after 0s [id=/providers/Microsoft.Authorization/roleDefinitions/43d0d8ad-25c7-4714-9337-8ba259a9fe05]
module.role_assignments.data.azurerm_management_group.management_groups_by_id_or_display_name["root"]: Read complete after 0s [id=/providers/Microsoft.Management/managementGroups/d1aa7522-0959-442e-80ee-8c4f7fb4c184]

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: unable to build authorizer: no Authorizer could be configured, please check your configuration
│ 
│   with provider["registry.terraform.io/hashicorp/azuread"],
│   on providers.tf line 59, in provider "azuread":
│   59: provider "azuread" {
│ 
╵
Releasing state lock. This may take a few moments...

@k8s-triage-robot
Copy link
Copy Markdown

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Mark this PR as fresh with /remove-lifecycle stale
  • Close this PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 20, 2026
@k8s-triage-robot
Copy link
Copy Markdown

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Mark this PR as fresh with /remove-lifecycle rotten
  • Close this PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/infra/azure Issues or PRs related to Kubernetes Azure infrastructure area/infra Infrastructure management, infrastructure design, code in infra/ area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants