Enhanced kubernetes version upgrades for workload clusters

[JTarasovic]

User Story

As an operator, I would like to be able to easily update the Kubernetes version of my workload clusters to be able to stay on top of security patches and new features.

Detailed Description

The procedure for updating the k8s version currently* is to copy the MachineTemplate for KCP, update KCP w/ new version and reference to new MachineTemplate which causes a rollout. Rinse and repeat for MachineDeployments.

• https://cluster-api.sigs.k8s.io/tasks/kubeadm-control-plane.html

Ideally, I'd be able to declare my intent to upgrade the workload cluster and that would be reconciled and rolled out for me.

Anything else you would like to add:

Discussed on 17 June 2020 weekly meeting.

/kind feature

[fabriziopandini]

This issue requires a certain degree of coordination across several components, so the first question in my mind is where to implement this logic.
I don't think this should go at cluster level, because cluster main responsibility is the cluster infrastructure, so what about assuming this should be implemented in separated extension (with it's own CRD/controller)?

[vincepri]

/milestone v0.4.0

We should revisit in v1alpha4 timeframe, probably needs a more detailed proposal

[CecileRobertMichon]

cc @rbitia

Ria, this might fit into your "cluster group" proposal?

[JTarasovic]

We have a relatively small (but growing) number of clusters so we're currently doing upgrades sort of manually. Conceptually, we think about our clusters in 3 streams - alpha, beta and stable - and roll out upgrades and configuration changes according to stream.

Our plan right now is to have common configuration for a stream in a CR (StreamConfig) w/ a controller. The StreamConfig controller would reconcile to ClusterConfigs based on label / annotation with its controller handling the actual cluster resource reconciliation (eg creation, k8s version upgrades, etc).¹

I don't think that it's CAPIs responsibility to implement all of that (or any) but if we can do some of the common stuff (version upgrades) here, that seems like it would be super valuable for the whole community. It also seems like the logic would be broadly applicable - copy template, update KCP, rollout, copy template, update MDs, rollout, profit².

¹Names are illustrative and not definitive. Something, something hard problems in Computer Science. ²Grossly over-simplified here for effect.

[vincepri]

Thanks for the extra context @JTarasovic, from everything I'm hearing here it might be worth considering some extra utilities/libraries/commands under clusterctl which could perform some variations of the concepts described above.

[seh]

Ideally, I'd be able to declare my intent to upgrade the workload cluster and that would be reconciled and rolled out for me.

I find that if I change the "spec.version" field in an existing KubeadmControlPlane object and apply the change, usually the controllers will upgrade my control plane, without me introducing a new (AWS)MachineTemplate. It sounds like that's not supposed to work, and yet it does—most of the time. Why is that?

[JTarasovic]

Does it actually change the version of the running cluster - eg kubectl get no -o wide shows the new version?

It did not in our experience. It would roll the control plane instances but they'd still be on the previous version.

[CecileRobertMichon]

This is how upgrading k8s version on control planes works currently: https://cluster-api.sigs.k8s.io/tasks/kubeadm-control-plane.html?highlight=rolling#how-to-upgrade-the-kubernetes-control-plane-version

Note that you might need to update the image as well if you are specifying the image to use in the machine template.

[seh]

Does it actually change the version of the running cluster - eg kubectl get no -o wide shows the new version?

Yes, it shows the new version there.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[vincepri]

/remove-lifecycle stale