cilium-operator can't start due to untolerated taint

[edwin-bruurs]

Environmental Info:
RKE2 Version:
rke2 version v1.32.8+rke2r1 (34960df)
go version go1.23.11 X:boringcrypto

Node(s) CPU architecture, OS, and Version:
Linux az1-cp-chmgm-9h66r 6.8.0-48-generic #48-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 14:04:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
3 Control planes (role: control-plane and etcd)
3 Worker nodes

Describe the bug:
Previously we created RKE2 clusters (via Rancher) using version 1.31.7. We recently started to create new clusters using 1.32.8. We use Cilium as the CNI. I've noticed that new clusters with 1.32.8 aren't starting as the Cilium Agents is not starting. This is caused by the agent waiting for the CRDs te be applied by the cilium operator. But since 1.32.8 the cilium operator is not starting due to untolerated taints on node-role.kubernetes.io/etcd and node.cloudprovider.kubernetes.io/uninitialized.

It looks like the Cilium Helm Release is updated in 1.18.000 to use more specific tolerations

Probably related to this commit: https://github.com/cilium/cilium/commit/a4c830dcc6284c675e2617359b1d8ac8a4e01911

Steps To Reproduce:
Install RKE2 via Rancher using control planes with the etcd and control-plane role. Afterwards cluster hangs in bootstrapping, as the cilium agent is not starting.

Expected behavior:
I would expect that the cilium-operator tolerates the taint node-role.kubernetes.io/etcd:NoExecute

Actual behavior:
The operator doesn't tolerate the node-role.kubernetes.io/etcd and thus the operator won't start and can't install CRDs preventing the cilium agent to start.

Workaround:
Add manually toleration to the cilium-operator for node-role.kubernetes.io/etcd and node.cloudprovider.kubernetes.io/uninitialized starts the operator, installs the CRDs and the agent will start afterwards.

[brandond]

untolerated taints on node-role.kubernetes.io/etcd and node.cloudprovider.kubernetes.io/uninitialized

The first taint is not one that we add, where is that coming from? Are you adding that yourself? I don't think it is expected that Cilium will tolerate random role taints.

The second indicates that the node has not yet been initialized by the cloud provider. What cloud provider are you using? Have you checked the CCM pod logs to see why it is not initializing the node?

[edwin-bruurs]

It looks like Rancher is setting the node-taints based on the role of the node. According /etc/rancher/rke2//etc/rancher/rke2/config.yaml.d/50-rancher.yaml the following taints are configured on the control-planes

"node-taint": [
  "node-role.kubernetes.io/control-plane:NoSchedule",
  "node-role.kubernetes.io/etcd:NoExecute"
],

The second taint should be removed by the CCM (rancher-vsphere-cpi), but I guess this container didn't start because no container network was available as the cilium-agent was not yet running, and was waiting on the cilium-operator to install the required CRDs for Cilium. As soon as I updated the tolerations on the cilium-operator, the CCM started and removed the node.cloudprovider.kubernetes.io/uninitialized. Will verify this early next week and will check the CCM logs accordingly.

[brandond]

The CCMs run with host network to avoid exactly this sort of chicken-and-egg problem with the CNI. Are you using the vsphere cpi chart that is bundled with RKE2?

[edwin-bruurs]

Yes we use the bundled one.

[brandond]

Can you confirm that you've not customized the chart values or daemonset to disable the vsphere CPI running with host network?

• https://github.com/rancher/rke2/blob/v1.32.8%2Brke2r1/charts/chart_versions.yaml#L35-L37
• https://github.com/rancher/rke2-charts/blob/main/charts/rancher-vsphere-cpi/rancher-vsphere-cpi/1.10.000/templates/daemonset.yaml#L115

[edwin-bruurs]

Just tested it again on a new cluster;

The cluster is now stuck on pre-bootstrap with the following pods;

cattle-system   cattle-cluster-agent-bootstrap-79cf67c4db-2nhh4                      0/1     Pending            0             2m31s
cattle-system   kube-api-auth-tsnr6                                                  1/1     Running            0             2m31s
kube-system     cilium-hrml5                                                         0/1     Running            0             2m25s
kube-system     cilium-operator-6dbf789945-8t5jd                                     0/1     Pending            0             2m25s
kube-system     cilium-operator-6dbf789945-mhnxm                                     0/1     Pending            0             2m25s
kube-system     etcd-workspace-prd-p27-eunl1-az1-cp-5wjq2-h8s47                      1/1     Running            0             2m32s
kube-system     helm-install-prometheus-operator-crds-cwqsh                          0/1     Completed          0             2m31s
kube-system     helm-install-rancher-vsphere-cpi-tk5cb                               0/1     Completed          0             2m31s
kube-system     helm-install-rancher-vsphere-csi-pq859                               0/1     Completed          0             2m31s
kube-system     helm-install-rke2-cilium-w68m5                                       0/1     Completed          0             2m31s
kube-system     helm-install-rke2-coredns-c9fwh                                      0/1     Completed          0             2m31s
kube-system     helm-install-rke2-metrics-server-t552p                               0/1     Pending            0             2m31s
kube-system     helm-install-rke2-runtimeclasses-pbzvf                               0/1     Pending            0             2m31s
kube-system     helm-install-rke2-snapshot-controller-crd-wgllt                      0/1     Pending            0             2m31s
kube-system     helm-install-rke2-snapshot-controller-h7ch4                          0/1     Pending            0             2m31s
kube-system     hubble-relay-57f4c565fb-wmsfs                                        0/1     Pending            0             2m25s
kube-system     kube-apiserver-workspace-prd-p27-eunl1-az1-cp-5wjq2-h8s47            1/1     Running            0             2m32s
kube-system     kube-controller-manager-workspace-prd-p27-eunl1-az1-cp-5wjq2-h8s47   1/1     Running            0             2m32s
kube-system     kube-scheduler-workspace-prd-p27-eunl1-az1-cp-5wjq2-h8s47            1/1     Running            0             2m32s
kube-system     rancher-vsphere-cpi-cloud-controller-manager-4xmfv                   0/1     CrashLoopBackOff   4 (46s ago)   2m26s
kube-system     rke2-coredns-rke2-coredns-86c455b944-px94q                           0/1     Pending            0             2m26s
kube-system     rke2-coredns-rke2-coredns-autoscaler-79677f89c4-l2z4d                0/1     Pending            0             2m26s
kube-system     vsphere-csi-controller-6b7b74d985-bpnxx                              0/7     Pending            0             2m26s
kube-system     vsphere-csi-controller-6b7b74d985-xpxxc                              0/7     Pending            0             2m26s
kube-system     vsphere-csi-controller-6b7b74d985-z28hx                              0/7     Pending            0             2m26s

These are the values we provide to the vSphere CPI (using Rancher in this case), so no changes on the host network or daemonset

rancher-vsphere-cpi:
  vCenter:
    credentialsSecret:
      generate: false
      name: rancher-vsphere-cpi-credentials
    datacenters: az1-k8s,az2-k8s,az3-k8s
    host: vcenter1.local
    labels:
      generate: true
      region: region
      zone: zone

This is the error in the vsphere-cpi logs;

unable to load configmap based request-header-client-ca-file: Get "https://10.40.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp 10.40.0.1:443: connect: network is unreachable

[brandond]

That would be the in-cluster endpoint for the kubernetes service. This should be handled by kube-proxy, but I don't see a kube-proxy pod present. Have you disabled it in an attempt to use the cilium kube-proxy replacement? If so, are you sure you've configured it correctly?

[edwin-bruurs]

Yes, we use the kube-proxy replacement from Cilium. which is configured with the following configuration

k8sServiceHost: 127.0.0.1
k8sServicePort: 6443
kubeProxyReplacement: true

But again in this case as the Cilium agent is not running (e.g. 0/1 Ready), I guess this proxy replacement is not yet working maybe while the cilium-agent is not yet ready?

This is the reason why the agent is not starting

time=2025-09-26T17:41:44.051367538Z level=info msg="Still waiting for Cilium Operator to register CRDs" module=agent.infra.k8s-synced-crdsync CRDs="[crd:ciliumclusterwidenetworkpolicies.cilium.io crd:ciliumloadbalancerippools.cilium.io crd:ciliumidentities.cilium.io crd:ciliumbgpclusterconfigs.cilium.io crd:ciliumcidrgroups.cilium.io crd:ciliumendpoints.cilium.io crd:ciliumnetworkpolicies.cilium.io crd:ciliumpodippools.cilium.io crd:ciliumbgpnodeconfigoverrides.cilium.io crd:ciliuml2announcementpolicies.cilium.io crd:ciliumnodes.cilium.io crd:ciliumbgppeeringpolicies.cilium.io crd:ciliumbgpadvertisements.cilium.io crd:ciliumbgpnodeconfigs.cilium.io crd:ciliumbgppeerconfigs.cilium.io]"

[brandond]

Hmm. Yeah, this is probably a combination of things - the upstream Cilium helm chart changed the tolerations in chart version 1.18.0, and you're trying to use the kube-proxy replacement which is stuck waiting on the operator to come up due to the tolerations change.

There are a few things you could do to address this, until we can patch the default tolerations:

• Don't enable tainting server nodes in your Rancher cluster config
• Correctly configure .operator.tolerations in your rke2-cilium HelmChartConfig to tolerate the taints added by Rancher
• Don't use the kube-proxy replacement

[edwin-bruurs]

Indeed, like already mentioned the second options works well. Ill also file a issue at Cilium, as at least voor the node.cloudprovider.kubernetes.io/uninitialized in combination with the kube-proxy replacement should be a pretty default setup I guess.

[edwin-bruurs]

@brandond, thank you for your effort to take a look into this.

[brandond]

Upstream change was made at:

• Change the default taints that Cilium tolerates to avoid deploying to a drained node cilium/cilium#40475