From 919b3e61d605f2aef486a427c81161d10e1d176d Mon Sep 17 00:00:00 2001 From: Scott Davidson <49713135+sd109@users.noreply.github.com> Date: Thu, 25 Jan 2024 10:57:15 +0000 Subject: [PATCH 1/6] Add section on deploying a local pulp --- README.rst | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/README.rst b/README.rst index 8679284..26208a4 100644 --- a/README.rst +++ b/README.rst @@ -242,6 +242,30 @@ This script will go through the process of performing the following tasks * openstack configuration * tempest testing +**Note**: When setting up a multi-node on a cloud which doesn't have access to test pulp (i.e. everywhere except SMS lab) a separate local pulp must be deployed. Before doing so, it is a good idea to make sure your seed VM has sufficient disk space by setting ``seed_disk_size`` in your ``terraform.tfvars`` to an appropriate value (100-200 GB should suffice). In order to set up the local pulp service on the seed, first obtain/generate a set of Ark credentials, then add the following configuration to ``etc/kayobe/environments/ci-multinode/stackhpc-ci.yml`` + +.. code-block:: console + + stackhpc_release_pulp_username: + stackhpc_release_pulp_password: !vault | + + + pulp_username: admin + pulp_password: + +You may also need to comment out many of the other config overrides in ``stackhpc-ci.yml`` such as ``stackhpc_repo_mirror_url`` plus all of the ``stackhpc_repo_*`` and ``stackhpc_docker_registry*`` variables. + +To create the local pulp as part of the automated deployment, add the following commands to the ``deploy-openstack.sh`` script in between ``kayobe seed service deploy`` and ``kayobe overcloud host configure``: + +.. code-block:: console + + kayobe seed service deploy --tags seed-deploy-containers --kolla-tags none -e deploy_containers_registry_attempt_login=false + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-repo-sync.yml + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-repo-publish.yml + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-container-sync.yml + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-container-publish.yml + + Accessing OpenStack ------------------- From 54286ae37a9dd92010981a7fe43c812950561ff8 Mon Sep 17 00:00:00 2001 From: Scott Davidson <49713135+sd109@users.noreply.github.com> Date: Thu, 25 Jan 2024 11:28:06 +0000 Subject: [PATCH 2/6] Address nits --- README.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.rst b/README.rst index 26208a4..6f2eb53 100644 --- a/README.rst +++ b/README.rst @@ -242,9 +242,9 @@ This script will go through the process of performing the following tasks * openstack configuration * tempest testing -**Note**: When setting up a multi-node on a cloud which doesn't have access to test pulp (i.e. everywhere except SMS lab) a separate local pulp must be deployed. Before doing so, it is a good idea to make sure your seed VM has sufficient disk space by setting ``seed_disk_size`` in your ``terraform.tfvars`` to an appropriate value (100-200 GB should suffice). In order to set up the local pulp service on the seed, first obtain/generate a set of Ark credentials, then add the following configuration to ``etc/kayobe/environments/ci-multinode/stackhpc-ci.yml`` +**Note**: When setting up a multinode on a cloud which doesn't have access to test pulp (i.e. everywhere except SMS lab) a separate local pulp must be deployed. Before doing so, it is a good idea to make sure your seed VM has sufficient disk space by setting ``seed_disk_size`` in your ``terraform.tfvars`` to an appropriate value (100-200 GB should suffice). In order to set up the local pulp service on the seed, first obtain/generate a set of Ark credentials using `this workflow `_, then add the following configuration to ``etc/kayobe/environments/ci-multinode/stackhpc-ci.yml`` -.. code-block:: console +.. code-block:: yaml stackhpc_release_pulp_username: stackhpc_release_pulp_password: !vault | @@ -253,7 +253,7 @@ This script will go through the process of performing the following tasks pulp_username: admin pulp_password: -You may also need to comment out many of the other config overrides in ``stackhpc-ci.yml`` such as ``stackhpc_repo_mirror_url`` plus all of the ``stackhpc_repo_*`` and ``stackhpc_docker_registry*`` variables. +You may also need to comment out many of the other config overrides in ``stackhpc-ci.yml`` such as ``stackhpc_repo_mirror_url`` plus all of the ``stackhpc_repo_*`` and ``stackhpc_docker_registry*`` variables which only apply to local pulp. To create the local pulp as part of the automated deployment, add the following commands to the ``deploy-openstack.sh`` script in between ``kayobe seed service deploy`` and ``kayobe overcloud host configure``: From 3b329aa8821e36db3ae291d1617b877a618a0613 Mon Sep 17 00:00:00 2001 From: Alex-Welsh Date: Thu, 25 Jan 2024 14:49:47 +0000 Subject: [PATCH 3/6] Automate pulp deployment --- README.rst | 12 ++---------- outputs.tf | 1 + templates/deploy-openstack.tpl | 8 ++++++++ variables.tf | 6 ++++++ 4 files changed, 17 insertions(+), 10 deletions(-) diff --git a/README.rst b/README.rst index 6f2eb53..b22969a 100644 --- a/README.rst +++ b/README.rst @@ -107,6 +107,7 @@ Generate Terraform variables: seed_vm_flavor = "general.v1.small" seed_disk_size = 100 + deploy_pulp = false multinode_flavor = "general.v1.medium" multinode_image = "Rocky9-lvm" @@ -255,16 +256,7 @@ This script will go through the process of performing the following tasks You may also need to comment out many of the other config overrides in ``stackhpc-ci.yml`` such as ``stackhpc_repo_mirror_url`` plus all of the ``stackhpc_repo_*`` and ``stackhpc_docker_registry*`` variables which only apply to local pulp. -To create the local pulp as part of the automated deployment, add the following commands to the ``deploy-openstack.sh`` script in between ``kayobe seed service deploy`` and ``kayobe overcloud host configure``: - -.. code-block:: console - - kayobe seed service deploy --tags seed-deploy-containers --kolla-tags none -e deploy_containers_registry_attempt_login=false - kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-repo-sync.yml - kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-repo-publish.yml - kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-container-sync.yml - kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-container-publish.yml - +To create the local Pulp as part of the automated deployment, set ``deploy_pulp`` to ``true`` in your ``terraform.tfvars`` file. Accessing OpenStack ------------------- diff --git a/outputs.tf b/outputs.tf index aaec76c..4e93c81 100644 --- a/outputs.tf +++ b/outputs.tf @@ -68,6 +68,7 @@ resource "local_file" "deploy_openstack" { seed_addr = openstack_compute_instance_v2.seed.access_ip_v4, ssh_user = var.ssh_user, deploy_wazuh = var.deploy_wazuh + deploy_pulp = var.deploy_pulp controller_hostname = openstack_compute_instance_v2.controller.*.name } ) diff --git a/templates/deploy-openstack.tpl b/templates/deploy-openstack.tpl index 9b7a59a..a811f0b 100644 --- a/templates/deploy-openstack.tpl +++ b/templates/deploy-openstack.tpl @@ -51,6 +51,14 @@ fi # Configure hosts kayobe control host bootstrap kayobe seed host configure +%{ if deploy_pulp } +# Deploy Pulp +kayobe seed service deploy --tags seed-deploy-containers --kolla-tags none -e deploy_containers_registry_attempt_login=false +kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-repo-sync.yml +kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-repo-publish.yml +kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-container-sync.yml +kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/pulp-container-publish.yml +%{ endif } kayobe overcloud host configure %{ if deploy_wazuh }kayobe infra vm host configure%{ endif } diff --git a/variables.tf b/variables.tf index 950c01c..0326edb 100644 --- a/variables.tf +++ b/variables.tf @@ -102,3 +102,9 @@ variable "deploy_wazuh" { type = bool default = false } + +variable "deploy_pulp" { + description = "Bool, whether or not to deploy Pulp." + type = bool + default = false +} From 02703012a431abfe5f5df399c723424af0c6c244 Mon Sep 17 00:00:00 2001 From: Alex-Welsh Date: Fri, 26 Jan 2024 10:15:01 +0000 Subject: [PATCH 4/6] Doing Scott's work for him --- README.rst | 50 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 33 insertions(+), 17 deletions(-) diff --git a/README.rst b/README.rst index b22969a..f1502e2 100644 --- a/README.rst +++ b/README.rst @@ -148,6 +148,12 @@ If `deploy_wazuh` is set to true, an infrastructure VM will be created that hosts the Wazuh manager. The Wazuh deployment playbooks will also be triggered automatically to deploy Wazuh agents to the overcloud hosts. +If `deploy_pulp` is set to true, a local pulp container will be deployed on the +seed node. This is mandatory for any multinode not running on SMS. Pulp can +sync a lot of data, so it is recommended that you ensure `seed_disk_size` is +greater than 150 when using this option. Local pulp deployments require +additional configuration, which is detailed below. + Generate a plan: .. code-block:: console @@ -220,18 +226,43 @@ These playbooks are tagged so that they can be invoked or skipped as required. F ansible-playbook -i ansible/inventory.yml ansible/configure-hosts.yml --skip-tags fqdn +The Ansible Control host should now be accessible with the following command: + +.. code-block:: console + + ssh $(terraform output -raw ssh_user)@$(terraform output -raw ansible_control_access_ip_v4) + +Deploy Pulp +----------- + +To set up a local pulp service on the seed, first obtain/generate a set of Ark credentials using `this workflow `_, then add the following configuration to ``~/src/kayobe-config/etc/kayobe/environments/ci-multinode/stackhpc-ci.yml``on the Ansible Control host. + +.. code-block:: yaml + + stackhpc_release_pulp_username: + stackhpc_release_pulp_password: !vault | + + + pulp_username: admin + pulp_password: + +Run the command below to automatically comment out the overrides in ``stackhpc-ci.yml`` for pointing to test pulp. + +.. code-block:: console + + sed -i -e 's/^resolv_/#resolv_/g' -e 's/^stackhpc_repo_/#stackhpc_repo_/g' -e 's/^stackhpc_include/#stackhpc_include/g' -e 's/^stackhpc_docker_registry:/#stackhpc_docker_registry:/g' ~/src/kayobe-config/etc/kayobe/environments/ci-multinode/stackhpc-ci.yml + Deploy OpenStack ---------------- Once the Ansible control host has been configured with a Kayobe/OpenStack configuration you can then begin the process of deploying OpenStack. -This can be achieved by either manually running the various commands to configures the hosts and deploy the services or automated by using `deploy-openstack.sh`, +This can be achieved by either manually running the various commands to configure the hosts and deploy the services or automated by using `deploy-openstack.sh`, which should be available within the homedir on your Ansible control host provided you ran `deploy-openstack-config.yml` earlier. If you choose to opt for automated method you must first SSH into your Ansible control host and then run the `deploy-openstack.sh` script .. code-block:: console - ssh $(terraform output -raw ssh_user)@$(terraform output -raw ansible_control_access_ip_v4) ~/deploy-openstack.sh This script will go through the process of performing the following tasks @@ -243,21 +274,6 @@ This script will go through the process of performing the following tasks * openstack configuration * tempest testing -**Note**: When setting up a multinode on a cloud which doesn't have access to test pulp (i.e. everywhere except SMS lab) a separate local pulp must be deployed. Before doing so, it is a good idea to make sure your seed VM has sufficient disk space by setting ``seed_disk_size`` in your ``terraform.tfvars`` to an appropriate value (100-200 GB should suffice). In order to set up the local pulp service on the seed, first obtain/generate a set of Ark credentials using `this workflow `_, then add the following configuration to ``etc/kayobe/environments/ci-multinode/stackhpc-ci.yml`` - -.. code-block:: yaml - - stackhpc_release_pulp_username: - stackhpc_release_pulp_password: !vault | - - - pulp_username: admin - pulp_password: - -You may also need to comment out many of the other config overrides in ``stackhpc-ci.yml`` such as ``stackhpc_repo_mirror_url`` plus all of the ``stackhpc_repo_*`` and ``stackhpc_docker_registry*`` variables which only apply to local pulp. - -To create the local Pulp as part of the automated deployment, set ``deploy_pulp`` to ``true`` in your ``terraform.tfvars`` file. - Accessing OpenStack ------------------- From a3a24769f5a3f18d63ce31674c3b8803a3a95b02 Mon Sep 17 00:00:00 2001 From: Alex-Welsh Date: Fri, 26 Jan 2024 10:22:09 +0000 Subject: [PATCH 5/6] Add pulp disclaimer --- README.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/README.rst b/README.rst index f1502e2..c874a7a 100644 --- a/README.rst +++ b/README.rst @@ -148,6 +148,10 @@ If `deploy_wazuh` is set to true, an infrastructure VM will be created that hosts the Wazuh manager. The Wazuh deployment playbooks will also be triggered automatically to deploy Wazuh agents to the overcloud hosts. +.. caution:: + + Local pulp deployment is a new feature and may not be stable + If `deploy_pulp` is set to true, a local pulp container will be deployed on the seed node. This is mandatory for any multinode not running on SMS. Pulp can sync a lot of data, so it is recommended that you ensure `seed_disk_size` is @@ -235,6 +239,10 @@ The Ansible Control host should now be accessible with the following command: Deploy Pulp ----------- +.. caution:: + + Local pulp deployment is a new feature and may not be stable + To set up a local pulp service on the seed, first obtain/generate a set of Ark credentials using `this workflow `_, then add the following configuration to ``~/src/kayobe-config/etc/kayobe/environments/ci-multinode/stackhpc-ci.yml``on the Ansible Control host. .. code-block:: yaml From f3626cae1765cd9bb00fe96a1fe7b48d474c75ce Mon Sep 17 00:00:00 2001 From: Alex-Welsh Date: Fri, 26 Jan 2024 11:26:39 +0000 Subject: [PATCH 6/6] Better pulp deployment automation --- README.rst | 6 ------ templates/deploy-openstack.tpl | 3 +++ 2 files changed, 3 insertions(+), 6 deletions(-) diff --git a/README.rst b/README.rst index c874a7a..8c9d5ee 100644 --- a/README.rst +++ b/README.rst @@ -254,12 +254,6 @@ To set up a local pulp service on the seed, first obtain/generate a set of Ark c pulp_username: admin pulp_password: -Run the command below to automatically comment out the overrides in ``stackhpc-ci.yml`` for pointing to test pulp. - -.. code-block:: console - - sed -i -e 's/^resolv_/#resolv_/g' -e 's/^stackhpc_repo_/#stackhpc_repo_/g' -e 's/^stackhpc_include/#stackhpc_include/g' -e 's/^stackhpc_docker_registry:/#stackhpc_docker_registry:/g' ~/src/kayobe-config/etc/kayobe/environments/ci-multinode/stackhpc-ci.yml - Deploy OpenStack ---------------- diff --git a/templates/deploy-openstack.tpl b/templates/deploy-openstack.tpl index a811f0b..b713335 100644 --- a/templates/deploy-openstack.tpl +++ b/templates/deploy-openstack.tpl @@ -48,6 +48,9 @@ if $(which apt 2>/dev/null >/dev/null); then sudo apt -y install uuid-runtime fi +# Edit config when deploying a local pulp repository +%{ if deploy_pulp } sed -i -e 's/^resolv_/#resolv_/g' -e 's/^stackhpc_repo_/#stackhpc_repo_/g' -e 's/^stackhpc_include/#stackhpc_include/g' -e 's/^stackhpc_docker_registry:/#stackhpc_docker_registry:/g' $KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/stackhpc-ci.yml %{ endif } + # Configure hosts kayobe control host bootstrap kayobe seed host configure