CAPO does not observe Member provisioning state when adding/deleting members of a pool

[jan-di]

/kind bug

What steps did you take and what happened:
Currently, when adding or deleting members from a loadbalancer pool, it waits until the loadbalancer is in provisoning state ACTIVE again, before considering the the process as done. With using amphorav2 as a loadbalancer provider, we noticed cases where the loadbalancer is not in a pending_* status, when a member is. This results in CAPO thinking the member is created/deleted successfully, while it may be actually stuck.

This should be the relevant piece of code:

cluster-api-provider-openstack/pkg/cloud/services/loadbalancer/loadbalancer.go

Line 721 in eefc873

if _, err := s.waitForLoadBalancerActive(lbID); err != nil {

This can lead to a more fatal situation:
As CAPO thinks members are correctly added, it continues to add new (control plane) nodes and remove old ones. We had cases were this resulted in a offline cluster, as all nodes were replaced by CAPI, but not correctly set in the Loadbalancer pool.

What did you expect to happen:
CAPO should check the provisioning status of the member resource and wait until it is active before going on.
In the fatal case, CAPO should either wait indefinitly for the member for becoming active, or should have some kind of failure/error. It should not continue the update process.

Anything else you would like to add:
I've already had a talk with @mnaser about this issue.

Environment:

• Cluster API Provider OpenStack version (Or git rev-parse HEAD if manually built): 0.12.3
• Cluster-API version: v1.10.2
• OpenStack version: idk
• Minikube/KIND version: not relevant
• Kubernetes version (use kubectl version): not relevant
• OS (e.g. from /etc/os-release): not relevant

[lentzi90]

Thanks for the well described issue!
Is this something you or @mnaser could work on? I am not sure I will have time for it at the moment.