Upgrade to Kubernetes 1.29.1, Update Dependencies, and Refresh URLs

README.md

@@ -22,11 +22,11 @@ The target audience for this tutorial is someone planning to support a productio
 
 Kubernetes The Hard Way guides you through bootstrapping a highly available Kubernetes cluster with end-to-end encryption between components and RBAC authentication.
 
-* [kubernetes](https://github.com/kubernetes/kubernetes) v1.21.5
-* [containerd](https://github.com/containerd/containerd) v1.4.4
-* [coredns](https://github.com/coredns/coredns) v1.8.3
-* [cni-plugins](https://github.com/containernetworking/plugins) v0.9.1
-* [etcd](https://github.com/etcd-io/etcd) v3.4.15
+* [kubernetes](https://github.com/kubernetes/kubernetes) v1.29.1
+* [containerd](https://github.com/containerd/containerd) v1.7.13
+* [coredns](https://github.com/coredns/coredns) v1.11.1
+* [cni-plugins](https://github.com/containernetworking/plugins) v1.4.0
+* [etcd](https://github.com/etcd-io/etcd) v3.5.12
 
 ## Labs
 

deployments/coredns.yaml

@@ -27,6 +27,13 @@ rules:
   - nodes
   verbs:
   - get
+- apiGroups:
+  - discovery.k8s.io
+  resources:
+  - endpointslices
+  verbs:
+  - list
+  - watch
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
@@ -96,10 +103,10 @@ spec:
         - key: "CriticalAddonsOnly"
           operator: "Exists"
       nodeSelector:
-        beta.kubernetes.io/os: linux
+        kubernetes.io/os: linux
       containers:
       - name: coredns
-        image: coredns/coredns:1.6.2
+        image: coredns/coredns:1.11.1
         imagePullPolicy: IfNotPresent
         resources:
           limits:

docs/01-prerequisites.md

@@ -201,7 +201,7 @@ The basic VM configuration process is the same for the 6 VM (you can also config
 
 You have to:
 
-* Install the [Ubuntu 18.04.4 LTS (Bionic Beaver) Server install image](https://releases.ubuntu.com/18.04/) on this VM.
+* Install the [Ubuntu 22.04.3 LTS Server install image](https://releases.ubuntu.com/22.04/) on this VM.
 
 * Configure the network interface (see the network architecture). Example of `/etc/netplan/00-installer-config.yaml` file if ens18 is the name of your private network interface (you need to change the IP address depending on the installed server):
 

docs/02-client-tools.md

@@ -9,9 +9,8 @@ The `cfssl` and `cfssljson` command line utilities will be used to provision a [
 On the **gateway-01** VM, download and install `cfssl` and `cfssljson`:
 
 ```bash
-wget -q --show-progress --https-only --timestamping \
-  https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/linux/cfssl \
-  https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/linux/cfssljson
+  wget -q --show-progress --https-only --timestamping https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64 -O cfssl
+  wget -q --show-progress --https-only --timestamping https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64  -O cfssljson
 ```
 
 ```bash
@@ -55,7 +54,7 @@ Runtime: go1.13
 The `kubectl` command line utility is used to interact with the Kubernetes API Server. On the **gateway-01** VM, download and install `kubectl` from the official release binaries:
 
 ```bash
-wget https://storage.googleapis.com/kubernetes-release/release/v1.21.5/bin/linux/amd64/kubectl
+wget https://storage.googleapis.com/kubernetes-release/release/v1.29.1/bin/linux/amd64/kubectl
 ```
 
 ```bash
@@ -77,7 +76,9 @@ kubectl version --client
 > Output:
 
 ```bash
-Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.21.5", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-17T11:41:22Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
+Client Version: v1.29.1
+Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
+
 ```
 
 Next: [Provisioning Compute Resources](03-compute-resources.md)

docs/07-bootstrapping-etcd.md

@@ -22,14 +22,14 @@ Download the official etcd release binaries from the [etcd](https://github.com/e
 
 ```bash
 wget -q --show-progress --https-only --timestamping \
-  "https://github.com/etcd-io/etcd/releases/download/v3.4.15/etcd-v3.4.15-linux-amd64.tar.gz"
+  "https://github.com/etcd-io/etcd/releases/download/v3.5.12/etcd-v3.5.12-linux-amd64.tar.gz"
 ```
 
 Extract and install the `etcd` server and the `etcdctl` command line utility:
 
 ```bash
-tar -xvf etcd-v3.4.15-linux-amd64.tar.gz
-sudo mv etcd-v3.4.15-linux-amd64/etcd* /usr/local/bin/
+tar -xvf etcd-v3.5.12-linux-amd64.tar.gz
+sudo mv etcd-v3.5.12-linux-amd64/etcd* /usr/local/bin/
 ```
 
 ### Configure the etcd Server

docs/08-bootstrapping-kubernetes-controllers.md

@@ -28,10 +28,10 @@ Download the official Kubernetes release binaries:
 
 ```bash
 wget -q --show-progress --https-only --timestamping \
-  "https://storage.googleapis.com/kubernetes-release/release/v1.21.5/bin/linux/amd64/kube-apiserver" \
-  "https://storage.googleapis.com/kubernetes-release/release/v1.21.5/bin/linux/amd64/kube-controller-manager" \
-  "https://storage.googleapis.com/kubernetes-release/release/v1.21.5/bin/linux/amd64/kube-scheduler" \
-  "https://storage.googleapis.com/kubernetes-release/release/v1.21.5/bin/linux/amd64/kubectl"
+  "https://storage.googleapis.com/kubernetes-release/release/v1.29.1/bin/linux/amd64/kube-apiserver" \
+  "https://storage.googleapis.com/kubernetes-release/release/v1.29.1/bin/linux/amd64/kube-controller-manager" \
+  "https://storage.googleapis.com/kubernetes-release/release/v1.29.1/bin/linux/amd64/kube-scheduler" \
+  "https://storage.googleapis.com/kubernetes-release/release/v1.29.1/bin/linux/amd64/kubectl"
 ```
 
 Install the Kubernetes binaries:
@@ -126,7 +126,7 @@ Documentation=https://github.com/kubernetes/kubernetes
 
 [Service]
 ExecStart=/usr/local/bin/kube-controller-manager \\
-  --address=0.0.0.0 \\
+  --bind-address=0.0.0.0 \\
   --cluster-cidr=10.200.0.0/16 \\
   --cluster-name=kubernetes \\
   --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \\
@@ -158,7 +158,7 @@ Create the `kube-scheduler.yaml` configuration file:
 
 ```bash
 cat <<EOF | sudo tee /etc/kubernetes/config/kube-scheduler.yaml
-apiVersion: kubescheduler.config.k8s.io/v1beta1
+apiVersion: kubescheduler.config.k8s.io/v1
 kind: KubeSchedulerConfiguration
 clientConnection:
   kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
@@ -291,11 +291,11 @@ In this section you will provision an Nginx load balancer to front the Kubernete
 
 ### Provision an Nginx Load Balancer
 
-Install the Nginx Load Balancer:
+Install the Nginx Load Balancer and stream library:
 
 ```bash
 sudo apt-get update
-sudo apt-get install -y nginx
+sudo apt-get install -y nginx libnginx-mod-stream 
 ```
 
 As **root** user, Create the Nginx load balancer network configuration:
@@ -348,12 +348,12 @@ curl --cacert ca.pem https://${KUBERNETES_PUBLIC_ADDRESS}:6443/version
 ```bash
 {
   "major": "1",
-  "minor": "21",
-  "gitVersion": "v1.21.5",
-  "gitCommit": "c96aede7b5205121079932896c4ad89bb93260af",
+  "minor": "29",
+  "gitVersion": "v1.29.1",
+  "gitCommit": "bc401b91f2782410b3fb3f9acf43a995c4de90d2",
   "gitTreeState": "clean",
-  "buildDate": "2020-06-17T11:33:59Z",
-  "goVersion": "go1.16.5",
+  "buildDate": "2024-01-17T15:41:12Z",
+  "goVersion": "go1.21.6",
   "compiler": "gc",
   "platform": "linux/amd64"
 }

docs/09-bootstrapping-kubernetes-workers.md

@@ -47,13 +47,13 @@ sudo swapoff -a
 
 ```bash
 wget -q --show-progress --https-only --timestamping \
-  https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.21.0/crictl-v1.21.0-linux-amd64.tar.gz \
-  https://github.com/opencontainers/runc/releases/download/v1.0.0-rc93/runc.amd64 \
-  https://github.com/containernetworking/plugins/releases/download/v0.9.1/cni-plugins-linux-amd64-v0.9.1.tgz \
-  https://github.com/containerd/containerd/releases/download/v1.4.4/containerd-1.4.4.linux-amd64.tar.gz \
-  https://storage.googleapis.com/kubernetes-release/release/v1.21.5/bin/linux/amd64/kubectl \
-  https://storage.googleapis.com/kubernetes-release/release/v1.21.5/bin/linux/amd64/kube-proxy \
-  https://storage.googleapis.com/kubernetes-release/release/v1.21.5/bin/linux/amd64/kubelet
+  https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.29.0/crictl-v1.29.0-linux-amd64.tar.gz \
+  https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64 \
+  https://github.com/containernetworking/plugins/releases/download/v1.4.0/cni-plugins-linux-amd64-v1.4.0.tgz \
+  https://github.com/containerd/containerd/releases/download/v1.7.13/containerd-1.7.13-linux-amd64.tar.gz \
+  https://storage.googleapis.com/kubernetes-release/release/v1.29.1/bin/linux/amd64/kubectl \
+  https://storage.googleapis.com/kubernetes-release/release/v1.29.1/bin/linux/amd64/kube-proxy \
+  https://storage.googleapis.com/kubernetes-release/release/v1.29.1/bin/linux/amd64/kubelet
 ```
 
 Create the installation directories:
@@ -72,9 +72,9 @@ Install the worker binaries:
 
 ```bash
 mkdir containerd
-tar -xvf crictl-v1.21.5-linux-amd64.tar.gz
-tar -xvf containerd-1.4.4-linux-amd64.tar.gz -C containerd
-sudo tar -xvf cni-plugins-linux-amd64-v0.9.1.tgz -C /opt/cni/bin/
+tar -xvf crictl-v1.29.0-linux-amd64.tar.gz
+tar -xvf containerd-1.7.13-linux-amd64.tar.gz -C containerd
+sudo tar -xvf cni-plugins-linux-amd64-v1.4.0.tgz -C /opt/cni/bin/
 sudo mv runc.amd64 runc
 chmod +x crictl kubectl kube-proxy kubelet runc
 sudo mv crictl kubectl kube-proxy kubelet runc /usr/local/bin/
@@ -96,7 +96,7 @@ Create the `bridge` network configuration file:
 ```bash
 cat <<EOF | sudo tee /etc/cni/net.d/10-bridge.conf
 {
-    "cniVersion": "0.4.0",
+    "cniVersion": "1.0.0",
     "name": "bridge",
     "type": "bridge",
     "bridge": "cnio0",
@@ -118,7 +118,7 @@ Create the `loopback` network configuration file:
 ```bash
 cat <<EOF | sudo tee /etc/cni/net.d/99-loopback.conf
 {
-    "cniVersion": "0.4.0",
+    "cniVersion": "1.0.0",
     "name": "lo",
     "type": "loopback"
 }
@@ -132,17 +132,10 @@ Create the `containerd` configuration file:
 ```bash
 sudo mkdir -p /etc/containerd/
 ```
+Set up containerd configuration to enable systemd Cgroups
 
 ```bash
-cat << EOF | sudo tee /etc/containerd/config.toml
-[plugins]
-  [plugins.cri.containerd]
-    snapshotter = "overlayfs"
-    [plugins.cri.containerd.default_runtime]
-      runtime_type = "io.containerd.runtime.v1.linux"
-      runtime_engine = "/usr/local/bin/runc"
-      runtime_root = ""
-EOF
+ containerd config default | sed 's/SystemdCgroup = false/SystemdCgroup = true/' | sudo tee /etc/containerd/config.toml
 ```
 
 Create the `containerd.service` systemd unit file:
@@ -194,6 +187,7 @@ authentication:
     clientCAFile: "/var/lib/kubernetes/ca.pem"
 authorization:
   mode: Webhook
+cgroupDriver: systemd
 clusterDomain: "cluster.local"
 clusterDNS:
   - "10.32.0.10"
@@ -220,12 +214,7 @@ Requires=containerd.service
 [Service]
 ExecStart=/usr/local/bin/kubelet \\
   --config=/var/lib/kubelet/kubelet-config.yaml \\
-  --container-runtime=remote \\
-  --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \\
-  --image-pull-progress-deadline=2m \\
   --kubeconfig=/var/lib/kubelet/kubeconfig \\
-  --network-plugin=cni \\
-  --register-node=true \\
   --v=2
 Restart=on-failure
 RestartSec=5
@@ -295,9 +284,21 @@ ssh root@controller-0 kubectl get nodes --kubeconfig admin.kubeconfig
 
 ```bash
 NAME       STATUS   ROLES    AGE   VERSION
-worker-0   Ready    <none>   15s   v1.21.5
-worker-1   Ready    <none>   15s   v1.21.5
-worker-2   Ready    <none>   15s   v1.21.5
+worker-0   Ready    <none>   15s   v1.29.1
+worker-1   Ready    <none>   15s   v1.29.1
+worker-2   Ready    <none>   15s   v1.29.1
+```
+
+> [!NOTE]
+> By default kube-proxy uses iptables to set up Service IP handling and load balancing. Unfortunately, it breaks our deployment and there's a hack to force Linux to run iptables even for bridge-only traffic:
+> 
+> Run this on all control and worker nodes. 
+
+```bash
+sudo modprobe br_netfilter
+echo "br-netfilter" >> /etc/modules-load.d/modules.conf
+sysctl -w net.bridge.bridge-nf-call-iptables=1
 ```
 
+
 Next: [Configuring kubectl for Remote Access](10-configuring-kubectl.md)

docs/10-configuring-kubectl.md

@@ -48,6 +48,49 @@ etcd-2               Healthy   {"health":"true"}
 etcd-0               Healthy   {"health":"true"}
 ```
 
+However component statuses are deprecated in Kubernetes 1.19 and later, so the recommended way to check cluster health is:
+```bash
+kubectl get --raw='/readyz?verbose'
+```
+
+> Output:
+
+```bash
+[+]ping ok
+[+]log ok
+[+]etcd ok
+[+]etcd-readiness ok
+[+]informer-sync ok
+[+]poststarthook/start-kube-apiserver-admission-initializer ok
+[+]poststarthook/generic-apiserver-start-informers ok
+[+]poststarthook/priority-and-fairness-config-consumer ok
+[+]poststarthook/priority-and-fairness-filter ok
+[+]poststarthook/storage-object-count-tracker-hook ok
+[+]poststarthook/start-apiextensions-informers ok
+[+]poststarthook/start-apiextensions-controllers ok
+[+]poststarthook/crd-informer-synced ok
+[+]poststarthook/start-service-ip-repair-controllers ok
+[+]poststarthook/rbac/bootstrap-roles ok
+[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
+[+]poststarthook/priority-and-fairness-config-producer ok
+[+]poststarthook/start-system-namespaces-controller ok
+[+]poststarthook/bootstrap-controller ok
+[+]poststarthook/start-cluster-authentication-info-controller ok
+[+]poststarthook/start-kube-apiserver-identity-lease-controller ok
+[+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok
+[+]poststarthook/start-legacy-token-tracking-controller ok
+[+]poststarthook/start-kube-aggregator-informers ok
+[+]poststarthook/apiservice-registration-controller ok
+[+]poststarthook/apiservice-status-available-controller ok
+[+]poststarthook/kube-apiserver-autoregistration ok
+[+]autoregister-completion ok
+[+]poststarthook/apiservice-openapi-controller ok
+[+]poststarthook/apiservice-openapiv3-controller ok
+[+]poststarthook/apiservice-discovery-controller ok
+[+]shutdown ok
+readyz check passed
+```
+
 List the nodes in the remote Kubernetes cluster:
 
 ```bash
@@ -58,9 +101,9 @@ kubectl get nodes
 
 ```bash
 NAME       STATUS   ROLES    AGE   VERSION
-worker-0   Ready    <none>   90s   v1.21.5
-worker-1   Ready    <none>   91s   v1.21.5
-worker-2   Ready    <none>   90s   v1.21.5
+worker-0   Ready    <none>   90s   v1.29.1
+worker-1   Ready    <none>   91s   v1.29.1
+worker-2   Ready    <none>   90s   v1.29.1
 ```
 
 Next: [Provisioning Pod Network Routes](11-pod-network-routes.md)

docs/12-dns-addon.md

@@ -7,20 +7,9 @@ In this lab you will deploy the [DNS add-on](https://kubernetes.io/docs/concepts
 Get the CoreDNS yaml:
 
 ```bash
-wget https://storage.googleapis.com/kubernetes-the-hard-way/coredns-1.8.yaml
+kubectl apply -f https://raw.githubusercontent.com/DushanthaS/kubernetes-the-hard-way-on-proxmox/master/deployments/coredns.yaml
 ```
 
-Edit the `coredns.yaml` file to change CoreDNS configuration to enable DNS resolution for external name:
-
-```bash
-sed '/.*prometheus :9153/a \ \ \ \ \ \ \ \ forward . /etc/resolv.conf' coredns.yaml
-```
-
-Deploy the `coredns` cluster add-on:
-
-```bash
-kubectl apply -f coredns.yaml
-```
 
 > Output:
 
@@ -52,7 +41,7 @@ coredns-699f8ddd77-gtcgb   1/1     Running   0          20s
 Create a `busybox` deployment:
 
 ```bash
-kubectl run --generator=run-pod/v1 busybox --image=busybox:1.28 --command -- sleep 3600
+kubectl run busybox  --image=busybox:1.28 --command -- sleep 3600
 ```
 
 List the pod created by the `busybox` deployment: