Magnum with Cluster API: Openstack volumes are not deleted after a cluster is deleted

[zonca]

See zonca/zonca.dev#10 (comment)

After deleting a Kubernetes cluster deployed with Magnum, several volumes are left behind.

[zonca]

@ana-v-espinoza you are right, even a cluster with the default configuration in the tutorial, which is just created and destroyed, leaves 3 openstack volumes behind.

It is using 7 volumes when running:

+--------------------------------------+---------------------------------------------------+--------+------+-----------------------------------------------------------------------+
| ID                                   | Name                                              | Status | Size | Attached to                                                           |
+--------------------------------------+---------------------------------------------------+--------+------+-----------------------------------------------------------------------+
| be880497-5f19-4e86-8da8-0532a089145a | pvc-8d577997-a2ba-47d6-9302-17b325adc407          | in-use |   10 | Attached to k8s2-nus6zvsuz4lo-default-worker-nvcvb-bmdv9 on /dev/vdc  |
| 608149d0-7499-4a58-9f44-bf2ee02fb3a0 | pvc-6f4732ab-ef50-4a0d-95ad-5b8b4fdae41e          | in-use |   10 | Attached to k8s2-nus6zvsuz4lo-default-worker-nvcvb-bmdv9 on /dev/vdd  |
| cd72ce86-258a-4de5-b3e2-ef9d36dfd2da | k8s2-nus6zvsuz4lo-control-plane-hfgzl-root        | in-use |   30 | Attached to k8s2-nus6zvsuz4lo-control-plane-hfgzl on /dev/vda         |
| 2660ab97-16d7-49e8-a110-f99084393c24 | pvc-38693fdf-11a7-457c-b9ff-b71fcec8ae84          | in-use |   10 | Attached to k8s2-nus6zvsuz4lo-default-worker-nvcvb-bmdv9 on /dev/vdb  |
| b342ddda-4185-4dca-ad55-82c29e444351 | k8s2-nus6zvsuz4lo-control-plane-tlnvv-root        | in-use |   30 | Attached to k8s2-nus6zvsuz4lo-control-plane-tlnvv on /dev/vda         |
| 573f3456-71aa-4220-9831-7e3df075a567 | k8s2-nus6zvsuz4lo-default-worker-nvcvb-bmdv9-root | in-use |   30 | Attached to k8s2-nus6zvsuz4lo-default-worker-nvcvb-bmdv9 on /dev/vda  |
| ad24d9fb-67f5-4376-8ada-a74d000e485b | k8s2-nus6zvsuz4lo-control-plane-t4ml8-root        | in-use |   30 | Attached to k8s2-nus6zvsuz4lo-control-plane-t4ml8 on /dev/vda         |
+--------------------------------------+---------------------------------------------------+--------+------+-----------------------------------------------------------------------+

but only 3 of those are actually kubernetes PVC.

And they are there after deletion:

o volume list
+--------------------------------------+---------------------------------------------------+-----------+------+-------------+
| ID                                   | Name                                              | Status    | Size | Attached to |
+--------------------------------------+---------------------------------------------------+-----------+------+-------------+
| be880497-5f19-4e86-8da8-0532a089145a | pvc-8d577997-a2ba-47d6-9302-17b325adc407          | available |   10 |             |
| 608149d0-7499-4a58-9f44-bf2ee02fb3a0 | pvc-6f4732ab-ef50-4a0d-95ad-5b8b4fdae41e          | available |   10 |             |
| 2660ab97-16d7-49e8-a110-f99084393c24 | pvc-38693fdf-11a7-457c-b9ff-b71fcec8ae84          | available |   10 |             |

[zonca]

I think this is due to the storage class configured by default with the Retain policy:

k get sc        
NAME                PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE     
default (default)   cinder.csi.openstack.org   Retain          WaitForFirstConsumer   true                   26m

[zonca]

Proposed solution

I think we always want the Delete reclaim policy, and then users can decide if they want to create volumes with Retain instead.
The best would be to change the Magnum Template.

However we could fix it user side:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cinder-delete
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: cinder.csi.openstack.org
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

k create -f storage_class_cinder_delete.yaml

make default storage class not default anymore:

    kubectl patch storageclass default \
  -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

so the storage classes are now:

cinder-delete (default)   cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   3m35s
default                   cinder.csi.openstack.org   Retain          WaitForFirstConsumer   true                   11m

We can also patch all PV in the monitoring namespace:

#!/bin/bash
NAMESPACE="monitoring-system"

# Retrieve all PVCs in the specified namespace
kubectl get pvc -n "$NAMESPACE" -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | while read -r pvc; do
  # Get the associated PV name
  pv=$(kubectl get pvc "$pvc" -n "$NAMESPACE" -o jsonpath='{.spec.volumeName}')
  
  # Check if PV name is retrieved
  if [ -n "$pv" ]; then
    echo "Patching PV: $pv"
    kubectl patch pv "$pv" -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'
  else
    echo "PVC $pvc does not have an associated PV."
  fi
done

[zonca]

This did not work, even if the reclaim policy is Delete, the volumes are still there after the cluster has been successfully deleted:

o volume list
+--------------------------------------+---------------------------------------------------+-----------+------+-------------+
| ID                                   | Name                                              | Status    | Size | Attached to |
+--------------------------------------+---------------------------------------------------+-----------+------+-------------+
| e8042414-2a6b-4b53-aeec-5288c0250722 | pvc-841ca071-9915-45f5-86e4-f3766c70eb09          | available |   10 |             |
| ca505890-61f8-4c7e-b3de-a83d86338980 | pvc-4ff0e23f-6020-43e4-bbaa-a684f1f028dd          | available |   10 |             |
| 23be0e94-14b6-4e80-b2fc-9e21e312e9c1 | pvc-e40f17c7-8331-41e7-9e65-936b0e3c38fe          | available |   10 |             |
| 3b3610a5-1a26-4480-a01a-43107acfd96b | pvc-da033b04-0671-4ef3-882c-d2cf4b17ce59          | available |   10 |             |
| ba5eae13-9168-4c3e-ad58-50742ba87a3b | k8s2-istioutkunml-default-worker-5rp7h-5tvwj-root | available |   30 |             |

I also noticed that the root disk of the worker is also not deleted, and this one has nothing to do with Kubernetes.

[zonca]

@julianpistorius would you have any suggestions? do you prefer I open a ticket via ACCESS?

[ana-v-espinoza]

CC: @julienchastang

Thanks Andrea, for this extra detective work.

I think a combination of patching the storage class to make the Reclaim Policy delete (as you've done) and deleting the namespaces where these PVCs exist might do the trick. In principle, deleting the namespaces should cascade down to any deployments, PVCs, PVs, and finally cinder/openstack to delete the volumes.

This does not address the root volumes persisting, however.

[zonca]

@ana-v-espinoza I just tested and I confirm it works.

so I put the necessary scripts under kubernetes_magnum/volumes, so the steps are:

cd kubernetes_magnum
kubectl create -f volumes storage_class_cinder_delete.yaml
kubectl patch storageclass default -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
bash volumes/set_volumes_delete.sh

then before deleting the cluster with Magnum we need to delete the monitoring system and all custom namespaces:

kubectl delete namespace monitoring-system
kubectl delete namespace jhub

It seems like a lot of work for something that probably all Magnum users want. I think it would be best to have the reclaim policy Delete baked into the Magnum images. @julianpistorius ?

Also, the root volume of the worker did not persist this time, not sure what is the difference.

[julianpistorius]

I just ran into the same issue yesterday: https://gitlab.com/jetstream-cloud/eot/tutorials/magnum-tutorial/-/issues/8#note_2465698607

I'll ask StackHPC folks.

[sd109]

Hi, sorry about this. The root cause here was an issue with one of the Cluster API management components on JS2 which should now be resolved. I created and deleted a test cluster an hour ago and all volumes (machine root volumes and additional PVCs) were correctly cleaned up after the cluster was deleted.

In case you're interested in more details, there is a process running on the JS2 CAPI management cluster called the cluster-api-janitor-openstack which is responsible for cleaning up these extra PVCs after a Magnum cluster is deleted and due to a bug in this component the clean up process had stalled and was not operating correctly. I'll propose a change to the janitor component upstream soon to try to prevent the issue from happening again.

[julianpistorius]

@sd109 from StackHPC knows what causes these problems, and has some answers that he'll add here and to #94.

[zonca]

thanks @sd109 , testing now

[zonca]

thanks @sd109, volumes are cleared.

However I noticed that the default reclaim policy is "Retain":

k get sc
NAME                PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
default (default)   cinder.csi.openstack.org   Retain          WaitForFirstConsumer   true                   8m42s
replicated-hdd      cinder.csi.openstack.org   Retain          WaitForFirstConsumer   true                   8m42s

k get pv -A
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                                                                                    STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-40587694-d67d-4e75-a024-48552041af38   10Gi       RWO            Retain           Bound    monitoring-system/storage-loki-stack-0                                                                                   default        <unset>                          6m29s
pvc-6c632aa5-dba7-4203-86fb-d9cbda12ae14   10Gi       RWO            Retain           Bound    monitoring-system/prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0           default        <unset>                          6m1s
pvc-a9a3fe63-8eab-4feb-ba23-654c56424e07   10Gi       RWO            Retain           Bound    monitoring-system/alertmanager-kube-prometheus-stack-alertmanager-db-alertmanager-kube-prometheus

Now, sometimes we would like to keep openstack volumes around. So I think the best behavior would be that the StorageClass has a default policy of Delete, so that most volumes are deleted. However, I can mark a volume as "Retain" (for example a large storage volume attached to NFS server) so I am sure that data are safe even in the event of cluster deletion.