Back to all use cases

AssessmentCross-Industry

The CloudStack database holds the cloud's state, so a backup nobody has restored is a guess

Infrastructure Platform Team

A typical engagement to check whether a CloudStack cloud can be recovered: the management database dumps, the state of the management server table, secondary storage and the system VMs, tested by restoring onto a spare environment.

CloudStack consulting

Overview

This is a typical engagement for an infrastructure team that runs one CloudStack zone for several internal teams and has never tested a recovery. The management servers are redundant, but everything depends on one MySQL database and on secondary storage holding templates and snapshots. AceMQ and a partner CloudStack specialist review what is backed up, test a restore on spare servers and write the recovery runbook.

Challenge

CloudStack's own upgrade guide tells operators to dump the cloud and cloud_usage databases before a change, and the same dumps are the starting point for any recovery. A dump that has never been restored may be incomplete or too old. Stale rows in the management server table can also stop a clustered upgrade from proceeding. And a restored database describes hosts, storage and system VMs that may no longer match what exists.

Environment

A pair of CloudStack management servers, one MySQL primary with a replica, KVM clusters with primary storage per cluster, NFS secondary storage, and per-zone system VMs and virtual routers.

Approach

AceMQ reads the backup jobs and their schedule, restores the latest dumps onto a spare database server and starts a management server against it in an isolated network. The engineers then compare what the restored database claims with what the real zone runs, and write down every difference and the order in which it would be reconciled.

Solution

  1. Lists every backup of the cloud and cloud_usage databases, its age, size and where it is stored
  2. Restores the latest dump onto a spare database server and starts a management server against it in an isolated network
  3. Compares hosts, primary and secondary storage and system VMs in the restored database with the real zone
  4. Checks the management server table for stale entries and records how they are cleared
  5. Reviews secondary storage capacity and the system VM states in each zone
  6. Writes the recovery runbook with the order of steps, the owner of each and the time each step took in the rehearsal

Outcome

The engagement delivers a backup inventory, a restore that has been rehearsed with its measured duration, a list of differences between a restored database and the live zone, a clean-up procedure for the management server table and a recovery runbook. The team stops relying on a dump it has never opened.

Technologies

Apache CloudStackMySQLKVM

Ready for an Apache CloudStack Health Check?

AceMQ's senior Apache CloudStack engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.