Allow projected volumes / Allow certain volumes - Support Azure Workload Identity

[BenjaminHerbert]

Describe the solution you'd like

I would like to be able to use the operator in AKS with workload identity enabled. This requires that additional volumes can be added on the fly to the pod definition.

I think this is a special case of #586.

Background

I am running an AKS cluster with workload identity enabled. This allows pods to use an annotated service account that can be used to access protected Azure resources.

The pod get s this label: azure.workload.identity/use: "true"

The service-account get's this annotation: azure.workload.identity/client-id: "<USER_ASSIGNED_CLIENT_ID

This requires that the pod gets a new volume, which looks like this:

  - name: azure-identity-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: api://AzureADTokenExchange
          expirationSeconds: 3600
          path: azure-identity-token

Due to some checks, the pod is being restarted as the operator determines that the volumes are different.

This hinders me to use Azure Workload Identity as authentication method e.g. in the azure-vm-agents plugin.

Describe alternatives you've considered

I would like to either

• have an exception like the hotfix for K8S 1.21, which ignores kube-api-access-<random-suffix> as volume name, or
• the possibility to configure which volumes should be ignored by a list of patterns which are ignored.

Additional context

Microsoft Entra Workload ID uses Service Account Token Volume Projection (that is, a service account), to enable pods to use a Kubernetes identity. A Kubernetes token is issued and OIDC federation enables Kubernetes applications to access Azure resources securely with Microsoft Entra ID, based on annotated service accounts.

• https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster

[BenjaminHerbert]

@brokenpip3 I would like to contribute here.

I think I could implement the "hotfix" approach analog to the kube-api-access hotfix.
Going the configuration route, I'd need some guidance.

[brokenpip3]

Hi Ben, thanks for offering your help.
The issue is clear to me like all the others that are part of the #586.
The fix is not so complex but it will require to rewrite a bit the checkForPodRecreation here.

The best way that I have in my mind to do it is to add for envs, volumes, annotations, and labels a list of "ignored" resources that will be ignored during the reconciliation. To do so we need:

• a new go struct that will contains these 4 lists
• handling the nill/empty case of the new go struct
• handling the ignore of the elements of that list for each of the category during the pod reconciliation
• helm chart values and template to handle the list

Another options is to decide the most used one (aks, eks, isto, etc) and put them there hardcoded and each time create a PR to add a new one, but this is way more ugly than the user lists I mention before.

Make sense to you?

[BenjaminHerbert]

I wonder, if it might be sufficient to ignore projected volumes with "serviceAccountToken" in sources in general?
This should also work for istio.

But if we have to touch it anyhow, maybe it's time to change the hardcoding to something configurable like you describe.

Another idea I had was to use whitelisting for the things we need to ensure and ignore the rest.
But I think this goes against the idea of an operator being owner of the resource. Am I right?

[brokenpip3]

I wonder, if it might be sufficient to ignore projected volumes with "serviceAccountToken" in sources in general?
This should also work for istio.

we can start with this, but for sure we also need to give to the users an optional list of annotations that needs to be ignored like other operators flow like the zalando pg operator

[BenjaminHerbert]

I first thought about putting it as a configuration on the operator, but if #706 comes to the conclusion that it's one operator and multiple Jenkins controllers (instances), this would be too limiting.

I'd see IgnoredAnnotations []string as part of JenkinsMaster struct.

Added here would be something like:

• IgnoredVolumes / IgnoredVolumeNames
• IgnoredAnnotations
• IgnoredEnvVars
• IgnoredLabels

I'd prefer this solution to the hardcoding after having thought about it.

[brokenpip3]

Added here would be something like:

* IgnoredVolumes / IgnoredVolumeNames

* IgnoredAnnotations

* IgnoredEnvVars

* IgnoredLabels

I'd prefer this solution to the hardcoding after having thought about it.

yup like I said initially this is will be the most clean and easy to maintain solution for this issue, thanks for speeding time trying to understand the best approach here :)

[psivananda]

any update on this? we are also blocked
we plan to use AWS Secret manager plugin and added IRSA for master for authentication and then master stuck at terminating loop

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

[BenjaminHerbert]

I won't find time to do this in the coming few months. If you want to take a shot at it, feel free to do so.

[brokenpip3]

anybody can work on this or other issues (maybe you can pair 🙂 ).
In general If the steps that we need to fix this are not clear let me know! 🙂 thanks