[CAPMox] Cloud-Init ISO injection fails on shared CephFS storage when deploying across multiple Proxmox nodes

[angelcruzlasso]

What steps did you take and what happened?

Description

When deploying a Cluster API management cluster using CAPMox (Cluster API Provider for Proxmox), VM provisioning fails if the cluster spans multiple Proxmox nodes — even when CephFS shared storage is used for Cloud-Init ISOs.

With a single node (the one hosting the template and management cluster), everything works fine.
When multiple nodes are allowed, CAPMox fails to inject the Cloud-Init ISO and VM provisioning stops with VMProvisionFailed.

Environment

Cluster API setup

• Guide followed: [Quick Start — Cluster API](https://cluster-api.sigs.k8s.io/user/quick-start)
• Infrastructure provider: CAPMox (Proxmox)
• Cluster API version: v1.10.x
• Kubernetes version (target cluster): v1.34.0
• Management cluster: kind (running inside a VM)
• OS: Ubuntu 22.04
• Shell: fish
• Network: 10.3.35.0/24 (university network, restricted internet access)

---

Proxmox environment

• Proxmox cluster: 4 nodes
  server-citic-5, server-citic-6, server-citic-7, server-citic-8

• Version: Proxmox VE 8.x

• Shared storage: CephFS (mounted at /mnt/pve/cephfs)

• VM storage pool for disks: RBD (vms)

• Template VM:

  • ID: 103
  • OS: Ubuntu 22.04 (built with [image-builder](https://image-builder.sigs.k8s.io/capi/providers/proxmox))
  • QEMU Guest Agent enabled
  • Cloud-init enabled
  • Disk located on vms pool
  • Cloud-init config stored in cephfs

Permissions setup:

• User: capmox@pve
• API Token: capmox@pve!capi
• Token permissions (confirmed in GUI):

• Administrator on /, /storage/cephfs, /storage/vms, etc.
• Propagate: true
• Shared storage CephFS enabled across all nodes.

---

Environment Variables

Used in Fish shell (.clusterctl.fish):

set -x CLUSTER_TOPOLOGY true
set -x PROXMOX_URL "https://10.3.35.98:8006"
set -x PROXMOX_TOKEN "capmox@pve!capi"
set -x PROXMOX_SECRET "••••••••••••••••••••••••••••••••"
set -x PROXMOX_SOURCENODE "server-citic-5"
set -x TEMPLATE_VMID 103
set -x CONTROL_PLANE_ENDPOINT_IP 10.3.35.150
set -x NODE_IP_RANGES "[10.3.35.151-10.3.35.170]"
set -x GATEWAY "10.3.35.1"
set -x IP_PREFIX 24
set -x DNS_SERVERS "[8.8.8.8,8.8.4.4]"
set -x ALLOWED_NODES "[server-citic-5,server-citic-6,server-citic-7,server-citic-8]"
set -x BRIDGE "vmbr0"
set -x BOOT_VOLUME_DEVICE "scsi0"
set -x PROXMOX_ISO_POOL "cephfs"
set -x PROXMOX_STORAGE_POOL "vms"
set -x PROXMOX_ISO_STORAGE_POOL "cephfs"

---

Reproduction steps

1. Create management cluster:

   kind create cluster

2. Initialize CAPI with CAPMox and IPAM:

   clusterctl init --infrastructure proxmox --ipam in-cluster

3. Generate and apply the cluster manifest:

   clusterctl generate cluster capi-quickstart \
     --kubernetes-version v1.34.0 \
     --control-plane-machine-count=3 \
     --worker-machine-count=3 > capi-quickstart.yaml
   
   kubectl apply -f capi-quickstart.yaml

---

Observed behavior

When ALLOWED_NODES includes multiple Proxmox nodes:

set -x ALLOWED_NODES "[server-citic-5,server-citic-6,server-citic-7,server-citic-8]"

Cluster creation fails during VM provisioning with this error:

unable to inject CloudInit ISO:
Post "https://10.3.35.98:8006/api2/json/nodes/server-citic-8/storage/cephfs/upload": EOF

The clusterctl describe cluster output shows:

│ ProxmoxMachine - VMProvisionFailed
│ unable to inject CloudInit ISO:
│ Post "https://10.3.35.98:8006/api2/json/nodes/server-citic-8/storage/cephfs/upload": EOF

Proxmox task log:

TASK ERROR: failed to stat '/var/tmp/pveupload-XXXXXXXXXXXXXX'

Despite CephFS being configured as shared storage, the upload request to the remote node fails.

---

Important findings

• When ALLOWED_NODES=[server-citic-5] (the same node that hosts the management cluster and the VM template), the cluster provisions successfully.

• When multiple nodes are allowed, CAPMox sometimes tries to clone the template from another node (e.g., server-citic-8) and fails when uploading the cloud-init ISO to CephFS.
• CephFS is shared and mounted on all nodes (/mnt/pve/cephfs, Shared: Yes).
• The token capmox@pve!capi has full Administrator privileges on all storages and paths.

This behavior indicates that the CAPMox controller is attempting to use the Proxmox API “upload” operation even for shared storage, leading to “failed to stat /var/tmp/pveupload-XXXX” errors when the API call is made to remote nodes.

---

What did you expect to happen?

Expected behavior

When using shared storage (CephFS):

• CAPMox should not attempt to re-upload the Cloud-Init ISO to the target node.
• Instead, it should write the ISO directly in the shared path (available to all cluster nodes) or reuse the template’s existing storage pool.

---

Logs / outputs

From kubectl describe cluster:

VMProvisionFailed
unable to inject CloudInit ISO:
Post "https://10.3.35.98:8006/api2/json/nodes/server-citic-8/storage/cephfs/upload": EOF

From Proxmox task viewer:

TASK ERROR: failed to stat '/var/tmp/pveupload-XXXXXXXXXXXXXX'

Working scenario (single node):

• No CloudInit injection errors.
• Cluster successfully reaches Provisioned state.

---

Possible root cause

In CAPMox’s VM creation flow (pkg/services/proxmoxmachine), the controller always issues an HTTP upload request to the selected node’s /api2/json/nodes/<target>/storage/<pool>/upload, even when the storage is marked as “shared: true”.

When Proxmox receives this call on a node different from where the /var/tmp/pveupload-* file exists, the stat operation fails because the temporary upload file only exists locally.

---

Suggested fix or enhancement

Before uploading Cloud-Init ISO:

1. Detect whether the selected storage pool is shared (storage.shared=true from /api2/json/storage).
2. If shared, use the same CephFS path without triggering an “upload” operation to another node.
3. Optionally, add a configuration parameter or environment variable to force CAPMox to reuse shared storage for cloud-init ISO injection.

Cluster API version

❯ kubectl get providers -A

NAMESPACE NAME AGE TYPE PROVIDER VERSION
capi-ipam-in-cluster-system ipam-in-cluster 3h5m IPAMProvider in-cluster v1.0.3
capi-kubeadm-bootstrap-system bootstrap-kubeadm 3h5m BootstrapProvider kubeadm v1.11.2
capi-kubeadm-control-plane-system control-plane-kubeadm 3h5m ControlPlaneProvider kubeadm v1.11.2
capi-system cluster-api 3h5m CoreProvider cluster-api v1.11.2
capmox-system infrastructure-proxmox 3h5m InfrastructureProvider proxmox v0.7.4

Kubernetes version

core in 🌐 100 in ~
❯ clusterctl version

clusterctl version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"a3139c21c0dbd6d7a930abbe6bd2050c60f328bc", GitTreeState:"clean", BuildDate:"2025-10-07T16:09:56Z", GoVersion:"go1.24.7", Compiler:"gc", Platform:"linux/amd64"}

core in 🌐 100 in ~
❯ kubectl version
Client Version: v1.34.1
Kustomize Version: v5.7.1
Server Version: v1.34.0

Anything else you would like to add?

• The issue does not occur when using only one node (allowedNodes=[server-citic-5]).
• CephFS storage is fully functional across all nodes.
• This problem prevents CAPMox from distributing control planes or workers across multiple Proxmox nodes, effectively breaking multi-node scheduling.

Label(s) to be applied

kind/bug
area/infrastructure-proxmox
triage/accepted
needs-investigation
/help

[struegamer]

same happens with a non cephfs shared storage.
My homelab looks like this:

NAS-ISCSI-Rack01 and NAS-ISCSI-Rack02 are iscsi targets, and those targets are exposed as LVM shared storages.
This works already nicely and no issues so far from Proxmox side.

Enabling cluster-api with CAPMOX as infra provider and setting up a PromoxMachineTemplate like this:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxMachineTemplate
metadata:
  name: control-plane-template
  namespace: default
spec:
  template:
    spec:
      disks:
        bootVolume:
          disk: scsi0
          sizeGb: 100
      format: qcow2
      full: true
      memoryMiB: 8192
      network:
        default:
          bridge: vmbr0
          vlan: 80
          model: virtio
      numCores: 4
      numSockets: 1
      sourceNode: pm-intel-06
      templateID: 101
      checks:
        skipQemuGuestAgent: true
        skipCloudInitStatus: true

and a ProxmoxCluster resource like this:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxCluster
metadata:
  name: mgmt-cluster-proxmox
  namespace: default
spec:
  allowedNodes:
  - pm-intel-01
  - pm-intel-02
  - pm-intel-03
  - pm-intel-04
  - pm-intel-05
  - pm-intel-06
  controlPlaneEndpoint:
    host: 192.168.80.5
    port: 6443
  dnsServers:
    - 192.168.80.1
  ipv4Config:
    addresses:
    - 192.168.80.50-192.168.80.75
    gateway: 192.168.80.1
    prefix: 24

results in this error:

E1103 17:38:05.693601       1 controller.go:347] "Reconciler error" err="failed to reconcile VM: failed to inject bootstrap data: unable to inject CloudInit ISO: Post \"https://pm-intel-01.infra.sourcecode.de:8006/api2/json/nodes/pm-intel-05/storage/isoimages/upload\": EOF" controller="proxmoxmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="ProxmoxMachine" ProxmoxMachine="default/control-plane-template-mfz68" namespace="default" name="control-plane-template-mfz68" reconcileID="93de0bc5-27c9-4238-a088-af8a090d642b"

the iso image storage is on NFS.

Therefore issue has a larger impact, not only on CephFS based Storage :)

API Token has full admin rights, so this is not the issue.

❯ kubectl get providers -A
NAMESPACE                     NAME                     AGE    TYPE                     PROVIDER      VERSION
cabpt-system                  bootstrap-talos          5h6m   BootstrapProvider        talos         v0.6.10
cacppt-system                 control-plane-talos      5h6m   ControlPlaneProvider     talos         v0.5.11
capi-ipam-in-cluster-system   ipam-in-cluster          5h6m   IPAMProvider             in-cluster    v1.0.3
capi-system                   cluster-api              5h6m   CoreProvider             cluster-api   v1.11.2
capmox-system                 infrastructure-proxmox   5h6m   InfrastructureProvider   proxmox       v0.7.5

clusterctl version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"Homebrew", GitTreeState:"clean", BuildDate:"2025-09-23T13:56:18Z", GoVersion:"go1.25.1", Compiler:"gc", Platform:"darwin/arm64"}

❯ kubectl version
Client Version: v1.32.7
Kustomize Version: v5.5.0
Server Version: v1.32.6+orb1

(running orbstack on mac as bootstrap cluster)

[wikkyk]

Tangentially related to #548

Thanks for the report. I added the enhancement label because while it is something that you'd expect to Just Work, it's also a bit of a new feature. CAPMOX just doesn't support this setup right now.
I would encourage you to work on a PR to improve this :)

[struegamer]

@wikkyk it's not related to #548 ide0 as cloud init device is available, it's just capmox which is misbehaving for the time being. It was working couple of versions before. and I think it's still working with the version Ionos is using for their commercial k8s offering and it's more a regression.
Potentially related bug issue on go-proxmox: luthermonson/go-proxmox#204 (comment)

and latest releases of capmox are missing the 0.2.3 go.mod bump (#554)

[barbirict]

I ran into the same issue with an NFS shared iso store.

Basically my temporary solution is to point the pve endpoint for CAPI to a haproxy instance, which then redirects the request to the correct node based on the path, elimantingthe upload problem.

My haproxy config:

frontend pve_frontend
    bind *:8006 ssl crt /etc/ssl/private/cert.io.pem
    bind [::]:8006 ssl crt /etc/ssl/private/cert.io.pem
    mode http
    option httplog
    timeout client 30s

    # ACLs match node name in API path
    acl is_a    path_beg -i /api2/json/nodes/a
    acl is_b  path_beg -i /api2/json/nodes/b
    acl is_c path_beg -i /api2/json/nodes/c
    acl is_d   path_beg -i /api2/json/nodes/d
    acl is_e  path_beg -i /api2/json/nodes/e
    acl is_f   path_beg -i /api2/json/nodes/f
    acl is_g   path_beg -i /api2/json/nodes/g
    acl is_h   path_beg -i /api2/json/nodes/h

    # Use ACLs to select backend
    use_backend pve_a    if is_a
    use_backend pve_b  if is_b
    use_backend pve_c if is_c
    use_backend pve_d   if is_d
    use_backend pve_e  if is_e
    use_backend pve_f   if is_f
    use_backend pve_g   if is_g
    use_backend pve_h   if is_h

    # Default backend (fallback)
    default_backend pve_f


backend pve_a
    mode http
    balance roundrobin
    server pve-a 10.20.100.21:8006 ssl verify none check

backend pve_b
    mode http
    balance roundrobin
    server pve-b 10.10.100.22:8006 ssl verify none check

backend pve_c
    mode http
    balance roundrobin
    server pve-c 10.10.100.23:8006 ssl verify none check

backend pve_d
    mode http
    balance roundrobin
    server pve-d 10.10.100.24:8006 ssl verify none check

backend pve_e
    mode http
    balance roundrobin
    server pve-e 10.10.100.25:8006 ssl verify none check

backend pve_f
    mode http
    balance roundrobin
    server pve-f 10.10.100.26:8006 ssl verify none check

backend pve_g
    mode http
    balance roundrobin
    server pve-g 10.10.100.27:8006 ssl verify none check

backend pve_h
    mode http
    balance roundrobin
    server pve-h 10.10.100.28:8006 ssl verify none check