Overview
This is a typical engagement for an operations team that runs a CloudStack cloud for internal tenants and has been told to move to the current long-term support line. The estate has two management servers, a MySQL database, KVM hosts in several clusters and a few system VMs per zone. AceMQ and a partner CloudStack specialist read the upgrade guides, inspect the estate and write the plan before anything is changed.
Challenge
CloudStack documents each upgrade in its own guide, from one release to the next, so an estate several releases behind has to read every guide on the path and note the steps that differ. The 4.22 guide asks for Java 17 as the default Java on the management server and on KVM hosts, for the new system VM template to be registered and READY before the upgrade, and for the database to be dumped first. From 4.20.3 and 4.22.1 a management server that finds a database upgrade is needed aborts if other management servers are still running. Any of these, missed on the night, turns a planned window into an outage.
Environment
Two CloudStack management servers behind a load balancer, a MySQL database for the cloud and cloud_usage schemas, KVM hosts on a mix of Ubuntu and RHEL-family releases, per-zone secondary storage, and a set of virtual routers and system VMs.
Approach
The engagement starts read-only. AceMQ lists the running CloudStack version, the Java version on every management server and KVM host, the operating system of each host, the system VM template in each zone and the state of the mshost table. It then reads the upgrade guide for each hop to the target and records what applies to this estate. Nothing in production is changed during the assessment.
Solution
- Records the running version, the Java version and the operating system of every management server and KVM host
- Reads the upgrade guide for every release between the running version and the LTS target and lists the steps that apply
- Checks that Java 17 is installable and can be the default on each host before the window
- Plans the system VM template registration and the wait for its READY state in each zone
- Writes the dump of the cloud and cloud_usage databases into the runbook, with a restore rehearsal on a spare server
- Orders the work: first management server, hosts and agents, remaining management servers, then system VMs and virtual routers, with a decision point and a rollback boundary after each
Outcome
The engagement delivers an inventory of versions and Java levels, the list of upgrade steps that apply to the estate, a window plan in the order above, a rehearsed database restore and a runbook the operations team can run. The team learns what must be true on every host before the first command is typed.