Upgrade to 4.22.0 - MOP For the Community: Ubuntu Upgrade Steps from 4.20.1 to 4.22.0

[tatay188]

MOP create on Jun 15 2025 by Omar F. Mendoza
MOP Update on Nov 11 2025 by Omar F. Mendoza

Giving Back to the community.
We are sharing to give something back to the community.
Use the MOP at your own risk

We create a MOP - Method of Procedure - for the Upgrade from 4.20.0 to 4.20.1 using Ubuntu.

We create a MOP - Method of Procedure - for the Upgrade from 4.20.1 to 4.22.0 using Ubuntu.

We Did not shutdown the MYSQL.
Take some time to come backup. We have had VMs running.

Simplified and It is based on the Official Documentation on : https://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/upgrade-4.20.html

STEP 1)
Go to the UI:
Go to IMAGES-- > TEMPLATES and click Register Template from URL
URL: http://download.cloudstack.org/systemvm/4.22/systemvmtemplate-4.22.0-x86_64-kvm.qcow2.bz2
Name: systemvm-kvm-4.22.0-x86_64
Description: systemvm-kvm-4.22.0-x86_64
Zone: Choose the zone where this hypervisor is used
Hypervisor: KVM
Format: QCOW2
Direct download: No/False (grayed button)
OS Type: Debian GNU/Linux 12 (64-bit) (or the highest Debian release number available in the dropdown)
Template Type: SYSTEM
Arch: X86_64
Extractable: no (grayed button)
Password Enabled: no (grayed button)
Public: no (grayed button)
Featured: no (grayed button)
Routing: no (grayed button)
HVM: yes/true (Button on Green)

Wait UNTIL THE Template Shows "Ready" - Click refresh from time to time.
ONCE THE SYSTEM TEMPLATE IS READY

STEP 2)

Go back to the Management server
On each management server proceed:

service cloudstack-management stop

service cloudstack-usage stop

cd /etc/cloudstack/management

cp config.json config.json.$(date +%Y-%m-%d-%H%M%S)
cp db.properties db.properties.$(date +%Y-%m-%d-%H%M%S)

SKIP THIS STEP: USE Only if STEP 1) fails or gives a problem use the following:
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt -m /mnt/secondary1 -u http://download.cloudstack.org/systemvm/4.22/systemvmtemplate-4.22.0-x86_64-kvm.qcow2.bz2 -h kvm -s -F

STEP 3)

Login to one of the MYSQL SERVERs and BACKUP:
mysqldump -u root -p -R cloud > cloud-backup_$(date +%Y-%m-%d-%H%M%S)
mysqldump -u root -p cloud_usage > cloud_usage-backup_$(date +%Y-%m-%d-%H%M%S)

STEP 4)
Go back to the Management server
On each management server proceed:

Replace Jammy with the Version of Ubuntu
cat >//etc/apt/sources.list.d/cloudstack.list<<EOF
deb http://download.cloudstack.org/ubuntu jammy 4.22
EOF

wget -4 -qO - http://download.cloudstack.org/release.asc | sudo apt-key add -

apt-get update
apt-get upgrade cloudstack-management -y

Configuration file '/etc/cloudstack/management/config.json'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** config.json (Y/I/N/O/D/Z) [default=N] ? N

For us the only difference is an additional Line, after "LoginFooter" "resetPasswordFooter": "",
the favicon='cloud.ico", we added manually, same we did for the last lines "notifyLatestCSVersion": true, "announcementBanner": {....}

When to choose N : We are not using the standard document links. If you don't need the document Links or have customized links then N.
When to choose Y : If you need the Document Links press Y, and copy your customizations for the logos and colors.

Configuration file '/etc/cloudstack/management/server.properties'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** server.properties (Y/I/N/O/D/Z) [default=N] ? N

Manually we added:
echo "" >>/etc/cloudstack/management/server.properties
echo "# The deployment mode for the extensions" >>/etc/cloudstack/management/server.properties
echo "extensions.deployment.mode=production" >>/etc/cloudstack/management/server.properties
echo "" >>/etc/cloudstack/management/server.properties
echo "# Thread pool configuration" >>/etc/cloudstack/management/server.properties
echo "#threads.min=10" >>/etc/cloudstack/management/server.properties
echo "#threads.max=500" >>/etc/cloudstack/management/server.properties

And restart the management servers:

systemctl restart cloudstack-management.service

apt-get upgrade cloudstack-usage

systemctl restart cloudstack-usage.service

STEP 5)

HYPERVISORS:
Login to the KVM HYPERVISORS
IN EACH Hypervisor do as follows:

cat >//etc/apt/sources.list.d/cloudstack.list<<EOF
deb http://download.cloudstack.org/ubuntu jammy 4.22
EOF

wget -4 -qO - http://download.cloudstack.org/release.asc | sudo apt-key add -

apt-get update

cd /etc/cloudstack/agent/
cp agent.properties agent.properties.$(date +%Y-%m-%d-%H%M%S)

service cloudstack-agent stop

apt-get upgrade cloudstack-agent -y

Configuration file '/etc/cloudstack/agent/agent.properties'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** agent.properties (Y/I/N/O/D/Z) [default=N] ? N

service cloudstack-agent start

added manually the line
echo "domr.scripts.dir=scripts/network/domr/kvm" >> /etc/cloudstack/agent/agent.properties

systemctl restart cloudstack-agent

STEP 6) SYSTEM VMs

Login as Admin to the Management server:
Go to >>Infrastructure>>>System VMs [Click the 3 verical-dots ] and select Patch System VM.

This time the Upgrade worked like a charm for the SystemVMs.

STEP 7) vROUTERS

OPTION 1:, Go to UI

INFRASTRUCTTURE >>>VIRTUAL ROUTERS
SELECT The first GROUP of 100 or 200 and SELECT PATCH ON THE UPPER MENU
ONCE COMPLETED
SELECT The first GROUP of 100 or 200 and SELECT FORCE REBOOT

OPTION 2:, Don't use it as it seems is BUGGY. It stops after the first number you have after "t" default is 10

On THE UI >>Infrastructure>>search for integration.api.port change to 8096

systemctl restart cloudstack-management.service
systemctl restart cloudstack-usage.service

the number after "t" is the number of vRourters that are going to be rebooted simultaneously.

nohup cloudstack-sysvmadm -d YourMySQLDB-IP -u YourUSer(cloud some times) -p YOURPASSWORD -r -t100> sysvm.log 2>&1 &

tail -f sysvm.log

On THE UI >>Infrastructure>>search for integration.api.port change to
integration.api.port 0

Go and restart all the management servers one more time:

systemctl restart cloudstack-management.service
systemctl restart cloudstack-usage.service

Note to the community. Be careful with your routing, as for some reason, the routes changed:
In our case we lost connectivity to the Secondary storage, and the routes were not backup.

No upgrades to FRR or Ubuntu - Jammy in our case as is the most stable. And Noble is buggy and does not have the support for the CEPH release we are using.

[shwstppr]

@tatay188 just a query why do you need to manually register/seed new system VM template? With recent releases of CloudStack, system VM templates are bundled with the RPM/DEB packages and will be automatically registered/seeded when you restart MS after the upgrade

[tatay188]

@shwstppr because we have a very short maintenance window. We will go with what we already validated that is going to work. We will try the bundled option on our lab first, and then the Production env next time.