Back to all use cases

ConsultingCross-Industry

Find out how long a restore really takes before an incident does it for you

Platform Engineering Team

A typical consulting engagement for a team whose MariaDB backups run every night but have never been restored: a timed restore to a scratch server and a binary log replay to a chosen point in time.

MariaDB consulting

Overview

A typical engagement starts with a team that runs MariaDB for an order system. Backups complete every night and the job reports success, but nobody has restored one since the system went live, and nobody knows how to recover to the minute before a bad deployment.

Challenge

A backup that completes is not a backup that restores. A full backup taken with mariadb-backup has to be prepared before it can be used, a point-in-time recovery needs the binary logs from the backup's recorded position onward, and the logs may have been purged by a retention setting shorter than the backup interval. The restore time depends on data size and storage and is unknown until it is measured.

Environment

A primary with replicas on virtual machines in the customer's data center, nightly full backups with mariadb-backup, binary logging enabled, and backup files copied to a separate storage system.

Approach

AceMQ engineers restore the latest full backup to a scratch server, run the prepare step, start the server and check the data against the source, timing each stage. They then replay binary logs from the position recorded with the backup to a chosen time and compare the result. The binary log retention, backup schedule and storage copy are reviewed against what the replay needs.

Solution

  1. Restore of the latest full backup to a scratch server, each stage timed
  2. Prepare step and start-up checked, with a data comparison against the source
  3. Binary log replay from the recorded position to a chosen time, with mariadb-binlog
  4. Binary log retention compared with the backup interval
  5. Restore time compared with the recovery objective the business has stated
  6. Runbook for restore and for point-in-time recovery, with a rehearsal schedule

Outcome

The engagement delivers a measured restore time, a replayed point-in-time recovery, a corrected retention and schedule where they fell short, and a runbook. The team repeats the rehearsal on a schedule, and AceMQ engineers can run the first one with it. Running the customer's production backups stays with the customer unless managed support is added.

Technologies

MariaDBLinux

Need MariaDB Architecture Guidance?

AceMQ's senior MariaDB engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.