backup-restore - Enable functional testing

This change:

- updates the cli/restore tool to make it delete the Deployments
  and StatefulSets Resources that will require a respawn from the
  sf-operator reconcile.
- adds a high-level functional test to simulate the backup/restore
  process.
- removes provious in-depth backup and restore role in order to only
  keep the high-level test.
- updates the CLI documenation

Change-Id: I385c83c959f8bc06097b1415f7711097889be79e

Author: morucci

CHANGELOG.md

@@ -0,0 +1 @@
+doc/reference/CHANGELOG.md

CHANGELOG.md

@@ -32,7 +32,9 @@ import (
 
 	"github.com/spf13/cobra"
 
+	appsv1 "k8s.io/api/apps/v1"
 	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/client-go/kubernetes"
 	ctrl "sigs.k8s.io/controller-runtime"
 )
@@ -162,6 +164,33 @@ func restoreZuul(ns string, backupDir string, kubeClientSet *kubernetes.Clientse
 
 }
 
+func clearComponents(ns string, kubeContext string) {
+	ctrl.Log.Info("Removing components requiring a complete restart ...")
+
+	env := cliutils.ENV{
+		Cli: cliutils.CreateKubernetesClientOrDie(kubeContext),
+		Ctx: context.TODO(),
+		Ns:  ns,
+	}
+
+	for _, stsName := range []string{"zuul-scheduler", "zuul-executor", "zuul-merger", "nodepool-builder", "zookeeper"} {
+		cliutils.DeleteOrDie(&env, &appsv1.StatefulSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      stsName,
+				Namespace: env.Ns,
+			},
+		})
+	}
+	for _, depName := range []string{"zuul-web", "nodepool-launcher"} {
+		cliutils.DeleteOrDie(&env, &appsv1.Deployment{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      depName,
+				Namespace: env.Ns,
+			},
+		})
+	}
+}
+
 func restoreCmd(kmd *cobra.Command, args []string) {
 
 	// NOTE: Solution for restoring DB and Zuul require kubectl binary to be installed and configured .kube/config
@@ -191,6 +220,7 @@ func restoreCmd(kmd *cobra.Command, args []string) {
 	restoreZuul(ns, backupDir, kubeClientSet, kubeContext)
 	restoreSecret(ns, backupDir, kubeContext)
 	restoreDB(ns, backupDir, kubeClientSet, kubeContext)
+	clearComponents(ns, kubeContext)
 
 }
 

cli/cmd/restore.go

@@ -166,6 +166,10 @@ func (r *SFController) InstallTooling() {
 
 func (r *SFController) SetupConfigJob() bool {
 
+	// Get the resource version of the keystore password
+	zkp := apiv1.Secret{}
+	r.GetM("zuul-keystore-password", &zkp)
+
 	// This ensure we trigger the base secret creation job when the setting change
 	extraSettingsChecksum := "ns"
 	if r.cr.Spec.ConfigRepositoryLocation.ClusterAPIURL != "" || r.cr.Spec.ConfigRepositoryLocation.LogserverHost != "" {
@@ -179,7 +183,7 @@ func (r *SFController) SetupConfigJob() bool {
 		cmName                       = "zs-internal-tenant-reconfigure"
 		zsInternalTenantReconfigure  apiv1.ConfigMap
 		configHash                   = utils.Checksum([]byte(r.MkPreInitScript()))
-		internalTenantSecretsVersion = "1" + "-" + extraSettingsChecksum
+		internalTenantSecretsVersion = "1" + "-" + zkp.ResourceVersion + "-" + extraSettingsChecksum
 		needReconfigureTenant        = false
 		needCMUpdate                 = false
 	)

controllers/config.go

@@ -45,10 +45,10 @@ The current project status is: **Alpha - NOT PRODUCTION READY**
 * Level 2 - Seamless upgrades - **2/2**
     - Operator: ✅
     - Operands: ✅
-* Level 3 - Full Lifecycle - **1/5**
+* Level 3 - Full Lifecycle - **3/5**
     - SF 3.8.x migration ❌
-    - Backup: ❌
-    - Restore: ❌
+    - Backup: ✅
+    - Restore: ✅
     - Rolling deployments: ❌
     - Reconfiguration: ✅
 * Level 4 - Deep Insights - **1/3**

doc/README.md

@@ -0,0 +1,20 @@
+# Backup and Restore
+
+The sf-operator CLI provides commands to perform a backup and a restore of a deployment managed by the sf-operator.
+
+The [backup command](../reference/cli/index.md#backup) can be run periodically to perform a backup of a Software Factory deployment.
+The command should be couple with a proper backup system to safely store the backuped data.
+
+Restoring a backup must be done via the [restore command](../reference/cli/index.md#restore) and by following a specific process:
+
+1. Deploy a Software Factory CR resource without `ConfigLocation` and Zuul connections definition.
+2. Restore the backup with the `restore` command
+3. Apply your final Software Factory CR
+
+## The backup archive
+
+The archive contains:
+
+- Some k8s Secret resources (like the Zuul Keystore Secret and Zuul SSH private key Secret)
+- The Zuul SQL database content (history of builds)
+- The Zuul projects' private keys (the keys stored into Zookeeper and used to encrypt/decrypt in-repo Zuul Secrets)

doc/deployment/backup-restore.md

@@ -20,5 +20,6 @@ and managing a Software Factory Custom Resource through SF-Operator.
 1. [Add corporate CA certificates into the CA trust chain](./corporate-certificates.md)
 1. [Monitoring](./monitoring.md)
 1. [Logging](./logging.md)
+1. [Backup and Restore](./backup-restore.md)
 1. [Deleting a deployment](./delete.md)
 1. [Custom Resource Definitions reference](./crds.md)

doc/deployment/index.md

@@ -0,0 +1,70 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [in development]
+
+### Added
+
+- CLI: restore command and documentation.
+- Dev CLI - Add command "go run ./main.go dev getImagesSecurityIssues" to ease getting a small report of HIGH
+  and CRITICAL Security issues reported by quay.io on container images used by the sf-operator.
+
+### Changed
+
+- Zookeeper version bumped to 3.8.4
+- The Operator handles only one Route resource as a 'gateway' pod dispatches incoming connections.
+
+### Deprecated
+### Removed
+
+- The LogsServer CRD and controller. As there is no identified need for a proper CRD and Controller.
+
+### Fixed
+### Security
+
+- UBI9/Zookeeper image rebuid to address reported security issues
+
+## [v0.0.27] - 2024-03-27
+
+🐰🔔 Easter release 🐰🔔
+
+### Added
+
+- "Debug" toggle for fluent bit sidecars
+- A support for running zuul-executor component external to the cluster (see ADR#014).
+- The standalone deployment mode exits 1 when the reconcile is not possible after 300 seconds
+- A bundled YAML file containing information about container images used by the operator `controllers/libs/base/static/images.yaml`
+
+### Changed
+
+- zookeeper: update liveness and readyness probes to only check SSL access and remove superfluous Service resource called
+  zookeeper-headless.
+- nodepool: update version to 10.0.0
+- zuul: update version to 10.0.0
+- CLI: simplified `SF backup` options to streamline the backup process.
+
+### Deprecated
+### Removed
+### Fixed
+
+- nodepool-builder: fixed the log path configuration when using the fluent bit log forwarder, resulting in much less file access errors appearing in fluent bit logs.
+
+### Security
+
+## [v0.0.26] - 2024-03-08
+
+### Added
+
+- CLI: Add the `SF backup` subcommand. This subcommand dumps a Software Factory's most important data for safekeeping.
+
+### Changed
+### Deprecated
+### Removed
+### Fixed
+### Security
+
+## [alpha] - not released
+
+- Initial alpha version. Please consult the commit log for detailed information.
+- From now on all changes will be referenced into this changelog.

doc/reference/CHANGELOG.md

@@ -4,11 +4,11 @@ We provide a command to perform various actions related to the management of Sof
 deployments, beyond what can be defined in a custom resource manifest.
 
 
-1. [Installing the CLI](#installing-the-cli)
-1. [Global Flags](#global-flags)
-1. [Configuration File](#configuration-file)
-1. [Subcommands](#subcommands)
-  1. [Dev](#dev)
+- [Installing the CLI](#installing-the-cli)
+- [Global Flags](#global-flags)
+- [Configuration File](#configuration-file)
+- [Subcommands](#subcommands)
+  - [Dev](#dev)
     - [cloneAsAdmin](#cloneasadmin)
     - [create demo-env](#create-demo-env)
     - [create gerrit](#create-gerrit)
@@ -17,20 +17,20 @@ deployments, beyond what can be defined in a custom resource manifest.
     - [run-tests](#run-tests)
     - [wipe gerrit](#wipe-gerrit)
     - [getImagesSecurityIssues](#getimagessecurityissues)
-  1. [Init](#init)
-  1. [Nodepool](#nodepool)
+  - [Init](#init)
+  - [Nodepool](#nodepool)
     - [configure providers-secrets](#configure-providers-secrets)
     - [create openshiftpods-namespace](#create-openshiftpods-namespace)
     - [get builder-ssh-key](#get-builder-ssh-key)
     - [get providers-secrets](#get-providers-secrets)
-  1. [Operator](#operator)
-  1. [SF](#sf)
-    1. [backup](#backup)
-    1. [bootstrap-tenant](#bootstrap-tenant)
-    1. [configure TLS](#configure-tls)
-    1. [restore](#restore)
-    1. [wipe](#wipe)
-  1. [Zuul](#zuul)
+  - [Operator](#operator)
+  - [SF](#sf)
+    - [backup](#backup)
+    - [bootstrap-tenant](#bootstrap-tenant)
+    - [configure TLS](#configure-tls)
+    - [restore](#restore)
+    - [wipe](#wipe)
+  - [Zuul](#zuul)
     - [create auth-token](#create-auth-token)
     - [create client-config](#create-client-config)
 
@@ -418,14 +418,9 @@ The following subcommands can be used to manage a Software Factory deployment an
 
 #### backup
 
-The `backup` subcommand lets you dump a Software Factory's most important files for safekeeping,
-most important files, such as:
+The `backup` subcommand lets you dump a Software Factory's most important files for safekeeping.
 
-- MariaDB Zuul database copy
-- secrets backup
-- Zuul project private keys
-
-To create a backup located in `/tmp/backup` directory of all important objects, run the following command:
+To create a backup located in `/tmp/backup` directory, run the following command:
 
 ```sh
 sf-operator SF backup --namespace sf --backup_dir /tmp/backup
@@ -437,6 +432,14 @@ Flags:
 |----------|------|-------|----|----|
 | --backup_dir | string | The path to the backup directory | no | - |
 
+The backup is composed of:
+
+- some relevant `Secrets` located in the deployment's namespace
+- the Zuul's SQL database
+- the Zuul's project's keys as exported by [zuul-admin export-keys](https://zuul-ci.org/docs/zuul/latest/client.html#export-keys)
+
+The backup directory content could be compressed and stored safely in a backup system.
+
 #### bootstrap-tenant
 
 Initialize a Zuul tenant's config repository with boilerplate code that define standard pipelines:
@@ -480,13 +483,7 @@ Flags:
 !!! warning
     The command requires to to have `kubectl` binary installed in the system
 
-The `restore` subcommand lets you restore:
-
-- Secrets
-- MariaDB Zuul database
-- Zuul project private keys
-
-that has been done by the `backup` command.
+The `restore` subcommand lets you restore a backup created with the `backup` command.
 
 For example:
 
@@ -500,6 +497,7 @@ Available flags:
 |----------|------|-------|----|----|
 | --backup_dir | string | The path to the backup directory to restore | yes | - |
 
+
 #### wipe
 
 The `wipe` subcommand can be used to remove all Software Factory instances in the provided namespace,