How to upgrade MariaDB properly using Replication Manager?

[PeterJanRoes]

MariaDB specifies a strict upgrade path for the primary and replicas. First all replicas should be upgraded to the newer version. After that a replica should be promoted to be the new primary. The old primary should not yet start replicating from the new primary, as the old primary is still on the old MariaDB version. The old primary should be upgraded first to the new MariaDB version and after that it can be joined as a replica to the new primary.

How is this upgrade process supported by Replication Manager? Of course it is trivial to upgrade all replicas while not changing the primary. But how can we switchover using Replication Manager to a replica and prevent replication from starting on the old primary? I have tried to find an option in the web interface, and have tried to find some documentation on how to upgrade a MariaDB master/slave cluster with Replication Manager. I could not yet find anything. Maybe I need to put the primary into Maintenance Mode before switching over? I cannot find any documentation on Maintenance Mode either. Can you point me into the right direction?

[svaroqui]

Hello Peter, the replication in MariaDB is compatible for upgrades and downgrades so just switchover should just work if not it is probably a bug, we have introduced an option to avoid switchover failover to lower version but the scenario you are after is not that one because we will switchover to upper version , indeed it looks like a lot more easy to do with 3 nodes clusters as you alway stay on replication topology with a valid replication. What MariaDB describe as best practice is to migrate to new major release via restoring dump in this case after the first switchover just take a backup using version 3.1 with not saving the system tables . if you detach the old master via reset slave all and activating the auto reseed settings a restart that standalone node. It should auto restore a backup on it with the correct GTID and reconfigure replication. Hope it helps let me know if something is not clear?

[svaroqui]

Maintenance just stop traffic from the proxies under control of repman

[PeterJanRoes]

Maintenance just stop traffic from the proxies under control of repman

Ok, thank you, that is clear now. Maybe some documentation can be added on this? Or did I miss it?

[PeterJanRoes]

About the proper way to upgrade MariaDB in a replication set up, I use the information on https://mariadb.com/kb/en/changing-a-replica-to-become-the-primary/. Here, at the top, it clearly states the following:

In MariaDB replication, a replica should be of a version same or newer than the primary. Because of this, one should first upgrades all replicas to the latest version before changing a replica to be a primary. In some cases one can have a replica to be of an older version than the primary, as long as one doesn't execute on the primary any SQL commands that the replica doesn't understand. This is however not guaranteed between all major MariaDB versions.

Emphasis is mine. And further below on the page (https://mariadb.com/kb/en/changing-a-replica-to-become-the-primary/#changing-the-old-primary-to-be-a-replica) it says:

Now you can upgrade the old primary to a newer version of MariaDB and then follow the same procedure to connect it as a replica.

So, a replica should always be of a newer (or the same) version than the primary, which will never be the case when the old primary is converted to a replica for a new already upgraded primary. The documentation notes that there might be cases that it is okay though. However, I do not want to risk the old primary to become corrupt. I do not see anything in the documentation that the preferred way is to restore from a backup. https://mariadb.com/kb/en/upgrading-between-major-mariadb-versions/ even notes that it is trivial to upgrade a single MariaDB instance, no backups needed.

So, I am looking for this option to be able to not join the old primary to the new primary after a switchover. The option you described above:

we have introduced an option to avoid switchover failover to lower version but the scenario you are after is not that one because we will switchover to upper version

looks like what we want, but indeed only in the other direction, to prevent replicating from a newer primary. Maybe the best way would be to add a specific action to switchover without rejoining the old primary, so a user can choose what he wants.

[svaroqui]

Why not just stopping your master after the ugrade of the slaves in such case from or out of the replication-manager api and let the failover automatic be triggered than once failed just upgrade binaries

In any case i report that if you follow minor release upgrade on regular basic the old master auto rejoin transparently .if you get case of incompatible version where it's not working please let us know

[svaroqui]

Indeed there is path that we could avoid having extra trx on the old master by FTWRL, wait propagation and shutdown but it should be cover via SHUTDOWN WAIT FOR ALL REPLICAS; if you are using api to shutdown

func (server *ServerMonitor) Shutdown() error {
	if server.Conn == nil {
		return errors.New("No database connection pool")
	}
	cluster := server.ClusterGroup
	cmd := "SHUTDOWN"
	if server.DBVersion.IsMariaDB() && server.DBVersion.Major >= 10 && server.DBVersion.Minor >= 4 && server.IsMaster() {
		cmd = "SHUTDOWN WAIT FOR ALL SLAVES"
	}
	_, err := server.Conn.Exec(cmd)
	if err != nil {
		cluster.LogModulePrintf(cluster.Conf.Verbose, config.ConstLogModGeneral, "TEST", "Shutdown failed %s", err)
		return err
	}
	return nil
}

[svaroqui]

Also for our case of docker services it is true that the service need to be re-provision to trigger the upgrade so we will transform this to have different api call like rolling-restart-for-upgrade

For each replica

• provision service to set MARIADB_AUTO_UPGRADE=1
• restart service
• provision service to set MARIADB_AUTO_UPGRADE=0

For leader

• set failover auto
• innodb_fast_shutdown=0
• provision service to set MARIADB_AUTO_UPGRADE=1
• shudown
• start service
• stop service
• provision service MARIADB_AUTO_UPGRADE=0
• start service
• switchback

Idealy change of docker version is auto detected and does not need this extra api call , but can be use for onpremise to return after the master shutdown