Magnum: Root volumes and docker volumes may be unnessesary?

[ana-v-espinoza]

Related to this comment in #93.

Is it possible to toggle the volume backed root drive for certain node groups? I can see that being useful for smaller VM flavors which have 20GB of storage, but for m3.mediums and above not only are we getting fewer storage (30GB vs 60GB), we're also using openstack volumes. This is also a potential problem for users with smaller allocations.

---

I suspected that this was what the --docker-volume-size argument was responsible for creating the root volumes, but create_cluster.sh sets that as 10GB, not the 30GB referenced in the above issue. It seems like that argument might not be doing anything. Perhaps related to us not setting a docker_volume_type?

$ openstack volume type list
+--------------------------------------+----------------+-----------+
| ID                                   | Name           | Is Public |
+--------------------------------------+----------------+-----------+
| f538e873-cd5b-45c0-99ed-1b8f55dbb0ac | replicated_hdd | True      |
| 895fb040-2aed-40a2-b2e6-a6daf2cdfa6d | __DEFAULT__    | True      |
+--------------------------------------+----------------+-----------+

Considering we haven't yet ran into any problems (that I've noticed, anyway) with the node's drives filling up, maybe an additional docker-volume isn't necessary, although a good option to know how to correctly use. For clarity, from the Magnum User Guide on docker-volume-size:

If specified, container images will be stored in a cinder volume of the specified size in GB. Each cluster node will have a volume attached of the above size. If not specified, images will be stored in the compute instance’s local disk.

---

The next candidate for where the root volumes are coming from is boot-volume-size, but neither create_cluster.sh nor the kubernetes-1-30-jammy cluster template set that, and the default boot-volume-size is 0, i.e. no boot volume.

Maybe the 30GB root volume we're observing is related to the way the ubuntu-jammy-kube-v1.30.4-240828-1653 image was created? An openstack image show on it does not present me with any information that immediately pops out to me.

[julianpistorius]

@ana-v-espinoza What happens when you don't pass --docker-volume-size to openstack coe cluster create at all?

[julianpistorius]

Alternatively, don't specify the boot volume size, and specify the Docker volume size appropriately?

[ana-v-espinoza]

Alternatively, don't specify the boot volume size, and specify the Docker volume size appropriately?

This is what we already do, unless I'm mistaken. See create_cluster.sh

---

What happens when you don't pass --docker-volume-size to openstack coe cluster create at all?

Let's find out...

First of all, make sure that the template doesn't have a docker_volume_size set by default:

➜ openstack coe cluster template show kubernetes-1-30-jammy
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                                                                                  |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| insecure_registry     | -                                                                                                                                                                      |
| labels                | {'monitoring_enabled': 'true', 'kube_dashboard_enabled': 'true', 'keystone_auth_enabled': 'false', 'capi_helm_chart_version': '0.10.1', 'octavia_provider': 'amphora'} |
| updated_at            | -                                                                                                                                                                      |
| floating_ip_enabled   | True                                                                                                                                                                   |
| fixed_subnet          | -                                                                                                                                                                      |
| master_flavor_id      | m3.small                                                                                                                                                               |
| uuid                  | 2d56f33a-713c-488a-b65f-47b625d832cd                                                                                                                                   |
| no_proxy              | -                                                                                                                                                                      |
| https_proxy           | -                                                                                                                                                                      |
| tls_disabled          | False                                                                                                                                                                  |
| keypair_id            | -                                                                                                                                                                      |
| public                | True                                                                                                                                                                   |
| http_proxy            | -                                                                                                                                                                      |
| docker_volume_size    | -                                                                                                                                                                      |
| server_type           | vm                                                                                                                                                                     |
| external_network_id   | public                                                                                                                                                                 |
| cluster_distro        | ubuntu                                                                                                                                                                 |
| image_id              | ubuntu-jammy-kube-v1.30.4-240828-1653                                                                                                                                  |
| volume_driver         | -                                                                                                                                                                      |
| registry_enabled      | False                                                                                                                                                                  |
| docker_storage_driver | -                                                                                                                                                                      |
| apiserver_port        | -                                                                                                                                                                      |
| name                  | kubernetes-1-30-jammy                                                                                                                                                  |
| created_at            | 2024-12-04T16:15:08+00:00                                                                                                                                              |
| network_driver        | calico                                                                                                                                                                 |
| fixed_network         | -                                                                                                                                                                      |
| coe                   | kubernetes                                                                                                                                                             |
| flavor_id             | m3.small                                                                                                                                                               |
| master_lb_enabled     | True                                                                                                                                                                   |
| dns_nameserver        | 1.1.1.1,8.8.8.8,8.8.4.4                                                                                                                                                |
| project_id            | 7146c507800e4e82be58e9035c58e80f                                                                                                                                       |
| hidden                | False                                                                                                                                                                  |
| tags                  | -                                                                                                                                                                      |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

After removing docker-volume-size from create_cluster.sh and running:

➜ bash create_cluster.sh
Request to create cluster dfe3e855-365f-4fff-8545-fdbc7eea8c17 accepted
Waiting for the cluster to become ready...
...
Time passed: 10 minutes. Current status: CREATE_COMPLETE
Cluster creation took 10 minutes.

➜ openstack volume list | grep vlm-test
| 489ceca0-d9fa-42bb-bbf3-a834bf862456 | pvc-b93e7695-9f4e-4cde-abf3-15665b9c833d              | in-use    |   10 | Attached to vlm-test-qhj2vbxysdb3-default-worker-l5r6b-7x7tj on /dev/vdc  |
| 3bc7dda9-e0ab-48d7-8fd3-f75ef68ae119 | pvc-6626ccf1-daa7-4690-add1-30aa54c89221              | in-use    |   10 | Attached to vlm-test-qhj2vbxysdb3-default-worker-l5r6b-7x7tj on /dev/vdd  |
| 39108345-f097-4efe-a48b-96c5412341ae | pvc-0c3466b5-c249-4b93-845b-c90f3460a55a              | in-use    |   10 | Attached to vlm-test-qhj2vbxysdb3-default-worker-l5r6b-7x7tj on /dev/vdb  |
| 87aa799f-0f13-426f-8559-63cc35077857 | vlm-test-qhj2vbxysdb3-default-worker-l5r6b-7x7tj-root | in-use    |   30 | Attached to vlm-test-qhj2vbxysdb3-default-worker-l5r6b-7x7tj on /dev/vda  |
| b4edfa43-0579-487c-a2e0-ee950fda52b8 | vlm-test-qhj2vbxysdb3-control-plane-9wbhz-root        | in-use    |   30 | Attached to vlm-test-qhj2vbxysdb3-control-plane-9wbhz on /dev/vda         |

Looks like the root volumes are still created and are still 30GB large. The other 3 PVCs are from the monitoring-system namespace.

➜ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                                                                                    STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-0c3466b5-c249-4b93-845b-c90f3460a55a   10Gi       RWO            Retain           Bound    monitoring-system/storage-loki-stack-0                                                                                   default        <unset>                          7m25s
pvc-6626ccf1-daa7-4690-add1-30aa54c89221   10Gi       RWO            Retain           Bound    monitoring-system/prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0           default        <unset>                          6m45s
pvc-b93e7695-9f4e-4cde-abf3-15665b9c833d   10Gi       RWO            Retain           Bound    monitoring-system/alertmanager-kube-prometheus-stack-alertmanager-db-alertmanager-kube-prometheus-stack-alertmanager-0   default        <unset>                          6m45s

Deleting the cluster, utilizing Andrea's patch to set the reclaim policy of the monitoring-system PVs to Delete:

➜ kubectl get pv -A | tail -n +2 | cut -f 1 -d " " > /tmp/pv.out # first get a list of PVs, just in case

➜ bash set_volumes_delete.sh

➜ kubectl delete ns monitoring-system
namespace "monitoring-system" deleted
➜ kubectl get all -n monitoring-system
No resources found in monitoring-system namespace.
➜ openstack volume list | grep -f /tmp/pv.out
# nothing returned :)

➜ bash delete_cluster.sh
Request to delete cluster vlm-test has been accepted.
Waiting for the cluster to be deleted...
Time passed: 0 minutes. Current status: DELETE_IN_PROGRESS.
Cluster successfully deleted.
Cluster deletion took 1 minutes.
➜ openstack volume list | grep vlm-test
# Interestingly, nothing is returned and the root volumes were successfully deleted

As Andrea describes in #93, the persistence of root volumes doesn't seem consistent.

[sd109]

The default root volume size comes from the admin-level default_boot_volume_{type,size} settings in the JS2 Magnum service which is not user visible. However, you should be able to set the boot_volume_{type,size} labels on the cluster at creation time in order to use something different from the default. Unfortunately this label currently gets applied to all node groups so it’s not possible to only configure boot from volume on some node groups and not others (though it would be relatively straightforward to add that configuration option to Magnum if necessary).

If you do choose to test some different volume settings then be aware that all cluster nodes require at least 20GB root disk and also that you'll likely encounter issues with etcd if you try to use HDD volumes for the VM root disks since etcd is pretty sensitive to disk latency and network-attached HDD storage is often not fast enough for etcd in our experience. If you do need to use HDD volumes for cluster VMs for any reason then there are some extra cluster config labels you can include to ensure that etcd gets allocated a small (e.g. 8GB) slice of SSD volume even if the VM root volumes are HDD backed. Unfortunately these options aren't well documented yet but you can see them in the code here.

The docker_volume_{type,size} arguments are not used at all by the magnum-capi-helm driver and are a remnant of the (now officially deprecated) Magnum Heat driver. Unfortunately the upstream Magnum documentation is painfully out of date and still has many such references to things which are only relevant to outdated drivers.

[zonca]

@ana-v-espinoza what do you think of Scott's answer above?

[ana-v-espinoza]

Hey Scott, Andrea,

Apologies for the late response. Unidata was going through a bit of a funding crisis for the last month or so which has now been resolved.

Thank you for the in-depth response. I imagine these are more JS2 questions than StackHPC ones:

Regarding etcd needing an SSD slide, I can only assume that the volume type I would have to request is the __DEFAULT__ one?

$ openstack volume type list
+--------------------------------------+----------------+-----------+
| ID                                   | Name           | Is Public |
+--------------------------------------+----------------+-----------+
| f538e873-cd5b-45c0-99ed-1b8f55dbb0ac | replicated_hdd | True      |
| 895fb040-2aed-40a2-b2e6-a6daf2cdfa6d | __DEFAULT__    | True      |
+--------------------------------------+----------------+-----------+

Are non-volume-backed instances (I think openstack refers to them as ephemeral) launched from HDD or SDD?

[zonca]

@ana-v-espinoza is this issue still relevant, is it useful if I look into it?

[ana-v-espinoza]

Andrea,

Thanks for the follow-up. I think I'll close this issue and re-open if it ever becomes relevant again.

Best,

ave