Graduate MachinePools from experimental to stable API

[mboersma]

What would you like to be added (User Story)?

As a production user of CAPI, I'm not comfortable working with "experimental" parts of the product, but I would like to take advantage of the native VM-grouping construct at my cloud provider.

Detailed Description

MachinePools have been part of the experimental API for years, but their API has been overall stable (recent backward-compatible changes for MachinePool Machines notwithstanding). Moving them out of exp/ will declare that MachinePools are stable, supported, and safe to use.

Anything else you would like to add?

The graduation criteria in the MachinePools proposal remains TBD. Informally, when the question "when can MPs graduate?" has been asked, the answer has generally been "after MachinePool Machines land."

MachinePool Machines have begun to land and should soon be largely complete. This seems like the time to consider graduating MachinePools, although there may be more context or specific // TODO items I'm not aware of. I'm hoping to identify those in this issue.

Label(s) to be applied

/kind feature

/cc @dtzar @Jont828

[killianmuldoon]

/triage accepted

[jayesh-srivastava]

@mboersma I would like to take this one up. Can we have discussions over subtaks/TODOs which can help me moving forward.

[fabriziopandini]

@mboersma
AFAIK as far as I remember we have still some to do left from @Jont828 PR in order to complete the implementation of the proposal and get feature parity with MD (this is at least what I remember from some thread on the PR and the discussion we had around it and the - divide and conquer from the original PR- )
e.g. label propagation, CC support, but also machines that can't be operated, probably more

It will be great to use this issue to track all this work

[vincepri]

It'd be also good to have a featureset comparison in the book to guide users when they should be using MachineDeployment or MachinePool

[Jont828]

Sounds good. #8842 is the last PR I currently have open related to MachinePool Machines. I think it makes sense to put a list of follow up tasks here once that gets wrapped up.

@vincepri We have a section in the CAPI book about differences in features but I suppose we can expand it.

[CecileRobertMichon]

#8858 should probably get resolved as well before we do this

[AndiDog]

Apart from some notes in #8858, here are some comments about the API.

• spec.replicas feels weird. Its meaning depends on what the infra provider (e.g. CAPA) and cloud provider (e.g. AWS) offer. For AWS ASGs, it's min/max/desired, where min/max are in AWSMachinePool.spec.{minSize,maxSize} and desired is in MachinePool.spec.replicas. What if a cloud provider doesn't have a "desired number of replicas" setting at all? I rather see these all in the <Infra>MachinePool type.
• spec.minReadySeconds is documented as "Minimum number of seconds for which a newly created machine instances should be ready." – As a user, I don't understand what the effect of this field is. Ready for that interval, and then? What does "ready" mean? Which component reports it as ready? If we don't improve the description, the value could be mistaken for the infra provider specific one (e.g. AWS ASG "instance warmup" in seconds). And since I wanted to understand this, I quickly searched in source code and didn't find this field used for machine pools – am I mistaken?
• https://cluster-api.sigs.k8s.io/tasks/experimental-features/machine-pools should document a "contract" – which fields should infra providers use and how? Which are required/optional to implement?

Other than that, I think stabilizing the API is a good idea – even a stable API can be evolved, for instance by adding new fields if we really run into major missing features.

[sbueringer]

And since I wanted to understand this, I quickly searched in source code and didn't find this field used for machine pools – am I mistaken?

This should be fixed via: #9837

Sounds like there's potential to improve the godoc for MD & MachinePool, but apart from that it seems to have a clear meaning

[dtzar]

Per the CAPI community call today as I understand it, the next steps are:

1. Immediately have someone PR to have Machinepool get deployed by default (without requiring feature flag being enabled). Push it also from alpha to beta status at this time.
2. Gather additional feedback after this change and consider setting a timeline with updated note(s) in the spec