CAPV + InClusterIPAM + Talos: IPs from addressesFromPools are not applied inside Talos (vSphere guestinfo.metadata vs Talos machine config)

[AkakievKD]

/kind bug
Title: Static IPs from addressesFromPools are not persisted on Talos VMs (CAPV passes IP via cloud-init/guestinfo, which Talos does not consume)

What steps did you take and what happened:
I deploy a Talos-based management/ workload cluster on vSphere via Cluster API + CAPV and allocate static IPs via CAPI IPAM (InClusterIPPool) referenced from VSphereMachineTemplate.spec.template.spec.network.devices[].addressesFromPools.

# IPAM pool
apiVersion: ipam.cluster.x-k8s.io/v1alpha2
kind: InClusterIPPool
metadata:
  name: pool
spec:
  addresses:
    - 192.168.163.150-192.168.163.160
  prefix: 24
  gateway: 192.168.163.2

---
# Infra: vSphere cluster + endpoint
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereCluster
metadata:
  name: cluster
spec:
  server: "192.168.163.158"
  controlPlaneEndpoint:
    host: 192.168.163.162
    port: 6443
  identityRef:
    kind: Secret
    name: vsphere-cluster

---
# CAPV machine template with addressesFromPools
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  name: master
spec:
  template:
    spec:
      cloneMode: fullClone
      datacenter: "Datacenter"
      datastore: "datastore1"
      folder: "testvm"
      resourcePool: "default"
      server: "192.168.163.158"
      template: "talos-cloud"
      network:
        devices:
          - networkName: "VM Network"
            # The key part: static addresses from IPAM
            addressesFromPools:
              - apiGroup: ipam.cluster.x-k8s.io
                kind: InClusterIPPool
                name: pool

---
# Talos control plane (Talos bootstrap provider)
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: TalosControlPlane
metadata:
  name: master
spec:
  version: v1.34.0
  replicas: 3
  infrastructureTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: VSphereMachineTemplate
    name: master
  controlPlaneConfig:
    controlplane:
      generateType: controlplane
      talosVersion: v1.11.1
      # Example network patch (two scenarios described below)
      strategicPatches:
        - |
          - op: replace
            path: /machine/network/interfaces
            value:
              - interface: eth0
                dhcp: false      # Scenario A (static in OS)
                vip:
                  ip: 192.168.163.162

Scenario A (Talos dhcp: false, relying on IPAM static):

On first boot, the VM may come up and the control plane can start, but after the first reboot the interface on Talos has no IP assigned anymore; the node becomes unreachable and the VIP cannot bind. Controller logs include messages like “connect: no route to host” and “no addresses were found for node”.

The IP remains allocated in CAPI/IPAM/CAPV objects, but it is not present inside the guest OS after reboot.

Scenario B (Talos dhcp: true + IPAM static):

On first boot, sometimes both a DHCP lease and the IPAM-provided address appear; sometimes only the IPAM address appears.

After a reboot, the Talos node keeps only the DHCP address; the IPAM-provided static IP disappears from the interface.

Talos with dhcp:false → after reboot the configured IP is gone on the interface.
Talos with dhcp:true → after reboot only the DHCP address remains; the IPAM static IP disappears.

What did you expect to happen:
When addressesFromPools is set, the IP address allocated by CAPI IPAM is consistently configured inside the guest OS and survives reboots for CAPV + Talos (i.e., static address reliably applied), or

If CAPV relies on cloud-init via guestinfo to apply network configuration (and the guest OS does not consume cloud-init), CAPV should clearly surface a warning (event/condition) and document that addressesFromPools will not configure networking in the guest OS for Talos, and that DHCP reservations or explicit static configuration in the Talos machine config must be used instead.

Anything else you would like to add:
From the CAPV code and docs, CAPV writes cloud-init user-data/metadata via vSphere guestinfo (guestinfo.userdata / guestinfo.metadata). That requires the guest OS to consume cloud-init to actually configure the network.

Talos Linux configures networking via its own machine configuration (e.g., machine.network.interfaces), and by default runs DHCP on interfaces with link; it does not consume cloud-init metadata for networking. So the IP set by CAPV/IPAM in guestinfo is not applied/persisted by Talos across reboots.

Related upstream discussion notes that CAPV currently places the IP in guestinfo.metadata, which Talos does not read, hence no static IP applied inside the VM

Environment:

• Cluster-api-provider-vsphere version: v1.14.0
• Kubernetes version: (use kubectl version): local meneged cluster 1.31.2
• OS (e.g. from /etc/os-release): debian 11.6 for local meneged cluster 1.31.2

[AkakievKD]

two cases

[AkakievKD]

my local cluster with vcenter and esxi

[chrischdi]

The fields on the API are clear on using cloudinit metadata:

https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/main/apis/v1beta1/types.go#L407-L411

The thing is, how does CAPV know that the OS is not using cloudinit so it should output a warning or error or so? IMHO it does not.

Not sure how this could work instead. Maybe the bootstrap provider has to do the IPAM contract stuff in that case, but I have no idea if that could even work (the CAPI enhancement or IPAM contract does not talk about this case).

Marking this as feature request instead of a bug, because the implementation was done for cloud init.

[AkakievKD]

in proxmox with nocloud image of talos (not vmware image of talos) it works correct in both cases

[chrischdi]

That has a different way of providing user data though because its a different provider.

Maybe it is fixable, but needs at least research. Taking a look at the proxmox provider how it does handle IPAM would be a first step to compare.

[chrischdi]

I did a quick look at the proxmox provider:

If I get it right it does its IP Assignment here:
https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/internal/service/vmservice/ip.go#L128
Not sure if it is reading it from there later and doing other things to it.
This uses a mechanism which simply does not exist (AFAIK) in CAPV.

I have no idea what Talos uses under the hood. (It obviously is not using cloud-init). If it is ignition, then it would require to implement support for that because currently it does only support cloud-init.

proxmox e.g. injects data like networking here: https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/internal/service/vmservice/bootstrap.go#L120 not sure if that's how it makes it work for talos.

But CAPV more or less only supports cloud-init for this use case currently.

[AkakievKD]

I think what's important is not how it creates this CRD, but how exactly it transfers the necessary data, which will then end up in guestinfo and possibly in userdata, and this will be fixed in the Talos machine config. And the Talos + Proxmox + IPAM bundle works correctly.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxCluster
metadata:
  name: cluster
spec:
  allowedNodes: ["test"]
  controlPlaneEndpoint:
    host: 192.168.163.61
    port: 6443
  dnsServers: ["8.8.8.8", "1.1.1.1"]
  ipv4Config: <--------------------------------------------------- this block create crd InClusterIPPool below
    addresses: ["192.168.163.51-192.168.163.60"]
    gateway: 192.168.163.2
    prefix: 24

# IPAM pool
apiVersion: ipam.cluster.x-k8s.io/v1alpha2
kind: InClusterIPPool
metadata:
  name: pool
spec:
  addresses:
    - 192.168.163.51-192.168.163.60
  prefix: 24
  gateway: 192.168.163.2

[AkakievKD]

After extensive research, I came to the following conclusions (if i did mistake let me know):

How Proxmox Provides IP Configuration
Obtaining Addresses from IPAM

In cluster-api-provider-proxmox, the controller allocates IP addresses for each network device by creating IPAddressClaim resources. After receiving an IP address, it stores it in machine.Status.IPAddresses and adds the ip_{device}_{address} tag to the VM https://raw.githubusercontent.com/ionos-cloud/cluster-api-provider-proxmox/main/internal/service/vmservice/ip.go#:~:text=. This way, IPAM provides static addresses to machines.

VM Data Injection (cloud-init/ignition)

CAPP uses the ISO injection mechanism. In the internal/service/vmservice/bootstrap.go file, the controller:

obtains bootstrap data (user-data) and network information (MAC address and dedicated IP);

When the MAC and IP are known, it calls the ISO injector;

If the format is cloud-init, it creates an ISO with the following files: user-data, meta-data, and network-config, where network-config is generated by the cloudinit.NewNetworkConfig function and contains YAML network: version: 2... ethernets: with static addresses, gateways, DNS, etc. https://raw.githubusercontent.com/ionos-cloud/cluster-api-provider-proxmox/main/internal/service/vmservice/bootstrap.go#:~:text=%2F%2F%20Inject%20userdata%20based%20on,if%20err%20%21%3D%20nil. The same code supports the Ignition format (used by Talos), which uses ignition.Enricher to add network configuration and attaches an empty network-config file for cloud-init https://raw.githubusercontent.com/ionos-cloud/cluster-api-provider-proxmox/main/internal/inject/inject.go#:~:text=%2F%2F%20Inject%20injects%20cloudinit%20userdata%2C,%7D. The ISO is mounted to the VM, and Talos reads the network configuration from the nocloud source (cloud-init) or from Ignition.

Network configuration template

The YAML template used for network-config is defined in pkg/cloudinit/network.go. It creates a network:version:2 document with interfaces, static addresses, route, DNS, and MTU: https://raw.githubusercontent.com/ionos-cloud/cluster-api-provider-proxmox/main/pkg/cloudinit/network.go#:~:text=%2F%2A%20network,end. This file is added to the ISO and loaded by Talos.

nocloud support in Talos

Talos documentation notes that when using nocloud images, Talos can retrieve machine configuration and network parameters from the ISO or SMBIOS. For Proxmox, it's recommended to create a cloud-init disk with the user-data, meta-data, and network-config files and click Regenerate Image. Talos will automatically apply static IPs, routes, and DNS on boot https://www.talos.dev/v1.11/talos-guides/install/cloud-platforms/nocloud/#:~:text=ds%3Dnocloud. This way, the network defined by IPAM persists across reboots.

Why does IPAM work in Proxmox but not in vSphere?

Data transfer mechanism. Proxmox uses an ISO disk with cloud-init files (user-data, meta-data, network-config) or Ignition. Talos supports these sources (nocloud/ignition) and reads network parameters. In VMware, CAPV transfers data via VMware guestinfo, which requires the use of cloud-init. Talos doesn't use cloud-init and doesn't read guestinfo.metadata.

Network configuration format. Proxmox passes YAML network: version: 2 in a separate file, network-config https://raw.githubusercontent.com/ionos-cloud/cluster-api-provider-proxmox/main/pkg/cloudinit/network.go#:~:text=%2F%2A%20network,end . Talos supports this format when booting from ISO. CAPV includes a similar YAML in guestinfo.metadata, which Talos ignores.

Support in Talos. Talos for VMware uses guestinfo.talos.config for bootstrap configuration but doesn't support reading cloud-init metadata. Therefore, the IPAM-allocated IP address isn't used. In Proxmox, Talos loads the configuration from the nocloud ISO, which is fully supported.

Warning in issue. The issue author suggests that CAPV add a warning or document that addressesFromPools will not configure the network for Talos, as Talos does not use cloud-init. The maintainer agreed that CAPV cannot detect whether a guest is using cloud-init, so it would be worth either documenting the limitation or developing support for other mechanisms (such as ISO injection), as implemented in Proxmox.

Conclusions

Proxmox + Talos supports IPAM through the use of nocloud/Ignition ISOs. Static IP addresses allocated by IPAM are placed in the network-config ISO and applied by Talos.

VMware vSphere + Talos uses guestinfo for data transfer. CAPV writes IP addresses from IPAM to guestinfo.metadata, expecting cloud-init to apply them. Talos does not use cloud-init, so static IP addresses are not applied. The network interface comes up via DHCP and loses the static IP after a reboot.

For IPAM to work correctly with Talos on VMware, you must either use nocloud ISO (which is not currently implemented in CAPV) or pass network parameters through other mechanisms supported by Talos (Ignition with network-config).

Thus, the difference in behavior is explained by different mechanisms for passing network settings and the lack of cloud-init metadata support in Talos. This is confirmed by both the providers' source code and the Talos documentation.

[chrischdi]

For IPAM to work correctly with Talos on VMware, you must either use nocloud ISO (which is not currently implemented in CAPV) or pass network parameters through other mechanisms supported by Talos (Ignition with network-config).

Thus, the difference in behavior is explained by different mechanisms for passing network settings and the lack of cloud-init metadata support in Talos. This is confirmed by both the providers' source code and the Talos documentation.

Yes, as I wrote above:

But CAPV more or less only supports cloud-init for this use case currently.

The way forward would be supporting ignition input too and adjust that instead of setting metadata.

[AkakievKD]

I think this functionality is critical. Is there any way to prioritize it?

[chrischdi]

Me and I think no other maintainer is able to commit to implement this soon. This more needs a volunteer to work on it.

However, while the above is one solution, I still question if this may be better part of the bootstrap provider instead.

Because the "inject into ignition config" thing is more or less modifying the data generated by the bootstrap provider.

[AkakievKD]

In any case, there must be some solution that is guaranteed to solve the problem described in the issue.