📖 Update autoscaling from zero enhancement proposal with support for platform-aware autoscale from zero

[aleskandro]

What this PR does / why we need it:

This PR updates the contract between the cluster-autoscaler Cluster API provider and the infrastructure provider's controllers that reconcile the Infrastructure Machine Template to support platform-aware autoscale from 0 in clusters consisting of nodes heterogeneous in CPU architecture and OS.

With this commit, the infrastructure providers implementing controllers to reconcile the status of their Infrastructure Machine Templates for supporting autoscale from 0 will be able to fill the status.nodeInfo stanza with additional information about the nodes.

The status.nodeInfo stanza has type corev1.NodeSystemInfo to reflect the same content, the rendered nodes' objects would store in their status field.

The cluster-autoscaler can use that information to build the node template labels kubernetes.io/arch and kubernetes.io/os if that information is present.

Suppose the pending pods that trigger the cluster autoscaler have a node selector or a requiredDuringSchedulingIgnoredDuringExecution node affinity concerning the architecture or operating system of the node where they can execute. In that case, the autoscaler will be able to filter the nodes groups options according to the architecture or operating system requested by the pod.

The users could already provide this information to the cluster autoscaler through the labels capacity annotation. However, there is no similar capability to support future labels/taints through information set by the reconcilers of the status of Infrastructure Machine Templates.

Which issue(s) this PR fixes

Related to #11961

/area provider/core

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign enxebre for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Welcome @aleskandro!

It looks like this is your first PR to kubernetes-sigs/cluster-api 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @aleskandro. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[JoelSpeed]

Seems very straightforward, no objection from me

[elmiko]

this generally makes sense to me, and i don't have any objections.

i would like to get some attention on the API field change though, just to make sure folks agree about the name and placement.

[aleskandro]

Thanks @elmiko. To give more context about the choice of naming and structure for everyone to discuss alternatives.

I couldn't consider the architecture as part of the capacity field because the values of keys in that stanza are expected to be a Quantity. I tried to trace back the reasoning the community did about the capacity field and I noticed that the Node objects' status field has the same field - capacity - with the same content we set in the Infrastructure Machine Template objects.

The Node objects' status field also has a nodeInfo field with the same name and type that I'm now proposing for the Infrastructure Machine Template objects.

Among the alternatives I was evaluating, this seemed the most reasonable to me.

[elmiko]

makes sense to me @aleskandro , no objection from me.

[chrischdi]

/hold

Especially for getting consensus on and resolve #11962 (comment)

[sbueringer]

@JoelSpeed If you find some time, PTAL :)

[sbueringer]

/ok-to-test

[sbueringer]

lgtm from my side. Pending resolution of #11962 (comment)

(I'll be on PTO for the next few weeks, but fine for me if there is consensus with other maintainers)

[aleskandro]

/label tide/merge-method-squash

[elmiko]

i am happy with the state this is in.

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 1e88b9ce31402b9352a67b7a6637e488da7436ac

[aleskandro]

/unhold

[fabriziopandini]

Just a few nits from my side.
Considering the type of changes we are introducing, I think we should consider addressing #12033 as integral part of the implementation.

Last nit, could you kindly also add a new line in the Implementation History down below

[fabriziopandini]

I would suggest to use NodeSystemInfo, so there is a more direct link with the corresponding struct in corev1 types.

[aleskandro]

Hi @fabriziopandini, the previous version was using NodeSystemInfo. I changed it due to #11962 (comment). cc @sbueringer

I'm ok with either way as far as we have consensus :).

[sbueringer]

I would favor NodeInfo because I think the field and struct name should be the same if there is no good reason to diverge. And I think the field name should be the same as in upstream

[fabriziopandini]

I'm ok with both options, but the consistency with corev1.Node in this case IMO could help

[aleskandro]

I agree with both your points, but leaning a bit more into @fabriziopandini's option to be consistent with corev1.Node.

[fabriziopandini]

TBH i struggle to follow, might be because this sentence enters into autoscaler implementation details.

What about simplifying by writing what takes precedence between capacity or node-info on the template vs annotations on MachineSet or MachineDeployment?

[aleskandro]

Changed to

The capacity.cluster-autoscaler.kubernetes.io/labels annotation takes precedence over other sources when conflicts occur.
For instance, if the kubernetes.io/arch label is specified in capacity.cluster-autoscaler.kubernetes.io/labels, its value supersedes that specified by node-info.cluster-autoscaler.kubernetes.io/architecture.

[k8s-ci-robot]

New changes are detected. LGTM label has been removed.

[aleskandro]

Just a few nits from my side. Considering the type of changes we are introducing, I think we should consider addressing #12033 as integral part of the implementation.

Hi @fabriziopandini , thanks for your review.

I could take #12033 and resolve it in a separate PR, if this sounds good to you.

Last nit, could you kindly also add a new line in the Implementation History down below

Done

[fabriziopandini]

Just to double check, are we sure we want to expose those types in a non-versioned package?
Also, given this is going to be part of the contract, shouldn't we use a package name that enforce this link (and that we can eventually expand with other contract related stuff)?

[aleskandro]

We were considering not to have these types in the cluster api repo, but delegating implementations to define their based on the contract defined here

[fabriziopandini]

I'm getting back to this because I'm still a little bit confused.

I was assuming that capacity.cluster-autoscaler.kubernetes.io and node-info.cluster-autoscaler.kubernetes.io serve a different pourpuse, so value cannot overlap (the example with both reporting arch is confusing).

Instead what can happen IMO is that each annotation overlaps with the corresponding struct in templates (capacity.cluster-autoscaler.kubernetes.io with status.capacity and node-info.cluster-autoscaler.kubernetes.io with status.nodeIndfo), and I was expecting this paragraph to define which of the two takes precedence

[aleskandro]

The issue here is that node-info and status.node(System)?Info will render as labels for the scheduler simulator in the autoscaler side. So either we deny capacity labels to specify labels like kubernetes.io/arch or we have to define which one takes precedence over the other.

The autoscaler and scheduler do not use status.nodeSystemInfo directly, but obey labels for pods that specify predicates for them