From 923bc4497b041099e79f1d3f7457238f53cb2985 Mon Sep 17 00:00:00 2001 From: Matt Crees Date: Thu, 25 Apr 2024 15:02:03 +0100 Subject: [PATCH 1/4] Add docs on handling Octavia certs --- source/operations_and_monitoring.rst | 146 +++++++++++++++++++++++++++ 1 file changed, 146 insertions(+) diff --git a/source/operations_and_monitoring.rst b/source/operations_and_monitoring.rst index f265d36..ce82e31 100644 --- a/source/operations_and_monitoring.rst +++ b/source/operations_and_monitoring.rst @@ -269,6 +269,152 @@ The images stored locally in the seed host can be seen using ``docker image ls`` Old and redundant images can be identified from their names and tags, and removed using ``docker image rm``. +Octavia ++++++++ + +Octavia uses mutual TLS to secure communication between the amphorae and +Octavia services. It uses a private CA to sign both client and server +certificates. We use the kolla-ansible built-in support for generating these +certificates: + +.. code-block:: console + + kayobe# kayobe kolla ansible run octavia-certificates + +This command will output certificates and keys in ``${KOLLA_CONFIG_PATH}/octavia-certificates`` + +Copy the relevant certificates into your kayobe-config: + +.. code-block:: console + + kayobe# cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia + kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem . + kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem . + kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.cert.pem . + kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.key.pem . + +Encrypt any files containing the keys: + +.. code-block:: console + + kayobe# ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault + Encryption successful + kayobe# ansible-vault encrypt server_ca.key.pem --vault-password-file ~/vault + Encryption successful + +Checking certificate expiry +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: console + + kayobe# ansible-vault decrypt client.cert-and-key.pem --vault-password-file ~/vault + Decryption successful + kayobe# openssl x509 -enddate -noout -in client.cert-and-key.pem + notAfter=Aug 12 10:45:35 2022 GMT + +Backing up the octavia-certificates directory +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the root of your kayobe-config checkout: + +.. code-block:: console + + kayobe# tools/backup-octavia-certificates.sh + +This will output an encrypted backup to ``$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar`` +Commit this file to store the backup. + +Restoring octavia-certificates directory when regenerating certificates +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the root of your kayobe-config checkout: + +.. code-block:: console + + kayobe# tools/restore-octavia-certificates.sh + +This will use the encrypted backup in ``$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar`` +to restore ``${KOLLA_CONFIG_PATH}/octavia-certificates``. This will allow you +to reuse the client CA. + +Rotating client.cert-and-key.pem +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This has a life time of 1 year. + +1) Follow the steps to restore octavia-certificates so you can reuse the client CA. +2) Make sure your config allows you to regenerate a certificate with the same common name + + .. code-block:: console + :caption: $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/index.txt.attr + + unique_subject = no + +3) Remove the old files relating to the client certificate: + + .. code-block:: console + + kayobe# rm $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/{client.cert-and-key.pem,client.csr.pem,client.cert.pem} + +4) Regenerate the certificates + + .. code-block:: console + + kayobe# kayobe kolla ansible run octavia-certificates + +5) Backup your octavia-certificates directory (see previous section). + +6) Copy your new certificate to the correct location: + + .. code-block:: console + + cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia + kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem . + kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem . + kayobe# ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault + Encryption successful + +7) Reconfigure octavia + + .. code-block:: console + + kayobe# kayobe overcloud service reconfigure -kt octavia + +8) Run tempest with the `octavia` test list to check it is working. + +9) Commit and push any changes. + +Rotating the CAs +~~~~~~~~~~~~~~~~ + +The CAs have a 10 year lifetime. Simply delete the relevant directory under +``$KOLLA_CONFIG_PATH/octavia-certificates/`` and regenerate it with: + + .. code-block:: console + + kayobe# kayobe kolla ansible run octavia-certificates + +Copy the relevant certificates into your kayobe-config. + +.. code-block:: console + + kayobe# cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia + kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem . + kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem . + kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.cert.pem . + kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.key.pem . + +Encrypt any files containing the keys. + +.. code-block:: console + + kayobe# ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault + Encryption successful + kayobe# ansible-vault encrypt server_ca.key.pem --vault-password-file ~/vault + Encryption successful + +Follow any instructions in the `upstream docs `_. + Backup of the OpenStack Control Plane ===================================== From df3de9b17abc92313d71752479d4d58ec603ea85 Mon Sep 17 00:00:00 2001 From: Matt Crees Date: Mon, 13 May 2024 15:17:44 +0100 Subject: [PATCH 2/4] Remove backups sections --- source/operations_and_monitoring.rst | 24 +++--------------------- 1 file changed, 3 insertions(+), 21 deletions(-) diff --git a/source/operations_and_monitoring.rst b/source/operations_and_monitoring.rst index ce82e31..94a01ef 100644 --- a/source/operations_and_monitoring.rst +++ b/source/operations_and_monitoring.rst @@ -312,30 +312,12 @@ Checking certificate expiry kayobe# openssl x509 -enddate -noout -in client.cert-and-key.pem notAfter=Aug 12 10:45:35 2022 GMT -Backing up the octavia-certificates directory -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In the root of your kayobe-config checkout: - -.. code-block:: console - - kayobe# tools/backup-octavia-certificates.sh - -This will output an encrypted backup to ``$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar`` -Commit this file to store the backup. - -Restoring octavia-certificates directory when regenerating certificates -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In the root of your kayobe-config checkout: +There is also support in Kolla-Ansible to check if certs will expire within a +given number of days: .. code-block:: console - kayobe# tools/restore-octavia-certificates.sh - -This will use the encrypted backup in ``$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/kolla/certificates/octavia-certificates-backup.tar`` -to restore ``${KOLLA_CONFIG_PATH}/octavia-certificates``. This will allow you -to reuse the client CA. + kayobe# kayobe kolla ansible run "octavia-certificates Rotating client.cert-and-key.pem ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 5ba1a674592a4ad0a96a136a85029252cfac0a95 Mon Sep 17 00:00:00 2001 From: Matt Crees Date: Mon, 13 May 2024 16:14:34 +0100 Subject: [PATCH 3/4] Update instructions to use leafcloud examples --- source/operations_and_monitoring.rst | 166 +++++++++++++++------------ 1 file changed, 95 insertions(+), 71 deletions(-) diff --git a/source/operations_and_monitoring.rst b/source/operations_and_monitoring.rst index 94a01ef..6713920 100644 --- a/source/operations_and_monitoring.rst +++ b/source/operations_and_monitoring.rst @@ -275,32 +275,12 @@ Octavia Octavia uses mutual TLS to secure communication between the amphorae and Octavia services. It uses a private CA to sign both client and server certificates. We use the kolla-ansible built-in support for generating these -certificates: +certificates. -.. code-block:: console - - kayobe# kayobe kolla ansible run octavia-certificates - -This command will output certificates and keys in ``${KOLLA_CONFIG_PATH}/octavia-certificates`` - -Copy the relevant certificates into your kayobe-config: - -.. code-block:: console - - kayobe# cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia - kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem . - kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem . - kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.cert.pem . - kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.key.pem . - -Encrypt any files containing the keys: - -.. code-block:: console - - kayobe# ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault - Encryption successful - kayobe# ansible-vault encrypt server_ca.key.pem --vault-password-file ~/vault - Encryption successful +We ensure that duplicate subject names are allowed, by setting +``unique_subject=no`` in +``etc/kayobe/kolla/octavia-certificates/client_ca/index.txt.attr``. This should +already be set. Checking certificate expiry ~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -319,83 +299,127 @@ given number of days: kayobe# kayobe kolla ansible run "octavia-certificates -Rotating client.cert-and-key.pem -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Rotating client certificates +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This has a life time of 1 year. +Octavia client certificates expire after a year, and should be rotated before +they expire. -1) Follow the steps to restore octavia-certificates so you can reuse the client CA. -2) Make sure your config allows you to regenerate a certificate with the same common name +.. note:: - .. code-block:: console - :caption: $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/index.txt.attr + Only the client certificate needs to be rotated annually. The server CA and + client CA both have 10 year expiries, and should not be rotated before this + since it would break the trust chain. - unique_subject = no +Decrypt all certificate files: -3) Remove the old files relating to the client certificate: +.. code-block:: console - .. code-block:: console - - kayobe# rm $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/{client.cert-and-key.pem,client.csr.pem,client.cert.pem} + git grep -l VAULT etc/kayobe/kolla/octavia-certificates | xargs ansible-vault decrypt --vault-password-file -4) Regenerate the certificates +Remove the existing client certificate and keys to ensure they are regenerated: - .. code-block:: console - - kayobe# kayobe kolla ansible run octavia-certificates +.. code-block:: console -5) Backup your octavia-certificates directory (see previous section). + rm etc/kayobe/kolla/octavia-certificates/client_ca/client.* -6) Copy your new certificate to the correct location: +Generate new certificates: - .. code-block:: console +.. code-block:: console - cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia - kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem . - kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem . - kayobe# ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault - Encryption successful + kayobe kolla ansible run octavia-certificates -ke node_custom_config=$KAYOBE_CONFIG_PATH/kolla/config -ke node_config=$KAYOBE_CONFIG_PATH/kolla -7) Reconfigure octavia +This is likely to modify several files, and possibly add some new ones. Add the +new files: - .. code-block:: console - - kayobe# kayobe overcloud service reconfigure -kt octavia +.. code-block:: console + + git add etc/kayobe/kolla/octavia-certificates/client_ca/ + +Encrypt the changed files that we are interested in: + +.. code-block:: console + + git grep -L VAULT etc/kayobe/kolla/octavia-certificates/client_ca | xargs ansible-vault encrypt --vault-password-file + git grep -L VAULT etc/kayobe/kolla/config/octavia/*.pem | xargs ansible-vault encrypt --vault-password-file + +Add and commit: -8) Run tempest with the `octavia` test list to check it is working. +.. code-block:: console + + git add etc/kayobe/kolla/octavia-certificates/client_ca/ + git add etc/kayobe/kolla/config/octavia/client.cert-and-key.pem etc/kayobe/kolla/config/octavia/client_ca.cert.pem + git commit -m "Octavia certs: rotate client certs" + +Discard other unnecessary changes: + +.. code-block:: console + + git status # check for anything unexpected! + git checkout etc/kayobe/kolla/octavia-certificates/ etc/kayobe/kolla/config/octavia + +Deploy the new client certs: -9) Commit and push any changes. +.. code-block:: console + + kayobe overcloud service deploy --kolla-tags octavia Rotating the CAs ~~~~~~~~~~~~~~~~ -The CAs have a 10 year lifetime. Simply delete the relevant directory under -``$KOLLA_CONFIG_PATH/octavia-certificates/`` and regenerate it with: +The CAs have a 10 year lifetime. When rotating these, you will also need to +generate new certificates. + +Decrypt all certificate files: + +.. code-block:: console + + git grep -l VAULT etc/kayobe/kolla/octavia-certificates | xargs ansible-vault decrypt --vault-password-file + +Remove the existing CAs where appropriate: + +.. code-block:: console + + rm -rf etc/kayobe/kolla/octavia-certificates/client_ca/ + rm -rf etc/kayobe/kolla/octavia-certificates/server_ca/ + +Generate new CAs and certificates: .. code-block:: console - - kayobe# kayobe kolla ansible run octavia-certificates -Copy the relevant certificates into your kayobe-config. + kayobe kolla ansible run octavia-certificates -ke node_custom_config=$KAYOBE_CONFIG_PATH/kolla/config -ke node_config=$KAYOBE_CONFIG_PATH/kolla + +Encrypt the changed files that we are interested in: + +.. code-block:: console + + git grep -L VAULT etc/kayobe/kolla/octavia-certificates/client_ca | xargs ansible-vault encrypt --vault-password-file + git grep -L VAULT etc/kayobe/kolla/config/octavia/*.pem | xargs ansible-vault encrypt --vault-password-file + +Add and commit: + +.. code-block:: console + + git add etc/kayobe/kolla/octavia-certificates/client_ca/ + git add etc/kayobe/kolla/octavia-certificates/server_ca/ + git add etc/kayobe/kolla/config/octavia/client.cert-and-key.pem etc/kayobe/kolla/config/octavia/client_ca.cert.pem + git commit -m "Octavia certs: rotate CAs" + +Discard other unnecessary changes: .. code-block:: console - kayobe# cd ${KAYOBE_CONFIG_PATH}/environments/$KAYOBE_ENVIRONMENT/kolla/config/octavia - kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client_ca.cert.pem . - kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/client.cert-and-key.pem . - kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.cert.pem . - kayobe# cp $KOLLA_CONFIG_PATH/octavia-certificates/client_ca/server_ca.key.pem . + git status # check for anything unexpected! + git checkout etc/kayobe/kolla/octavia-certificates/ etc/kayobe/kolla/config/octavia -Encrypt any files containing the keys. +Deploy the new client certs: .. code-block:: console - kayobe# ansible-vault encrypt client.cert-and-key.pem --vault-password-file ~/vault - Encryption successful - kayobe# ansible-vault encrypt server_ca.key.pem --vault-password-file ~/vault - Encryption successful + kayobe overcloud service deploy --kolla-tags octavia -Follow any instructions in the `upstream docs `_. +Follow any instructions in the `upstream docs +`_. Backup of the OpenStack Control Plane ===================================== From e24fd046484239463eeb79e333ef801363d02c2c Mon Sep 17 00:00:00 2001 From: Matt Crees Date: Mon, 13 May 2024 16:16:32 +0100 Subject: [PATCH 4/4] Fix indentation --- source/operations_and_monitoring.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/source/operations_and_monitoring.rst b/source/operations_and_monitoring.rst index 6713920..9190b4f 100644 --- a/source/operations_and_monitoring.rst +++ b/source/operations_and_monitoring.rst @@ -385,9 +385,9 @@ Remove the existing CAs where appropriate: Generate new CAs and certificates: - .. code-block:: console +.. code-block:: console - kayobe kolla ansible run octavia-certificates -ke node_custom_config=$KAYOBE_CONFIG_PATH/kolla/config -ke node_config=$KAYOBE_CONFIG_PATH/kolla + kayobe kolla ansible run octavia-certificates -ke node_custom_config=$KAYOBE_CONFIG_PATH/kolla/config -ke node_config=$KAYOBE_CONFIG_PATH/kolla Encrypt the changed files that we are interested in: