Back to all use cases

AssessmentEducation

Finding out what the failover would actually do before it had to

Online Learning Platform

Reviewing a replicated MariaDB estate behind MaxScale and finding stale reads, a failover script written for MySQL GTIDs, and tables with no primary key.

Overview

An online learning platform served course content and student progress from a MariaDB primary with several replicas behind MaxScale. Teachers had reported progress screens that showed old data after a bulk import, and nobody was sure the automated failover had ever been exercised end to end.

Challenge

Several problems sat together. The readwritesplit router had no replication lag limit, so a lagging replica kept receiving reads. Some large tables had no primary key, which turns every row event into a full scan on a replica under row-based replication. And the promotion script had been adapted from a MySQL runbook that assumes UUID-based GTIDs, which MariaDB does not use.

Environment

One MariaDB 10.11 primary and four replicas on managed virtual machines in a single cloud, with MaxScale 23 in front and Prometheus for monitoring.

Approach

AceMQ inspected runtime configuration and replication state directly, then ran a controlled failover in a staging copy to see what the script did. Findings were ranked by how likely each was to cause stale data or a failed promotion.

Solution

  1. Compared gtid_slave_pos, gtid_binlog_pos and gtid_current_pos across servers and checked gtid_strict_mode and domain IDs
  2. Identified tables with no primary key and quantified the replica apply cost of each under row-based replication
  3. Set a maximum replication lag in MaxScale readwritesplit and reviewed the MariaDB Monitor failover settings
  4. Ran a staged failover, which showed the promotion script choosing the wrong position, and rewrote it for MariaDB GTIDs
  5. Checked parallel replication settings against the batch window and transaction sizes
  6. Delivered a ranked findings report with a fix order and the checks to repeat after each change

Outcome

The customer got a clear list of what would have gone wrong in a real failover, and fixed the promotion script and the lag limit within two weeks. Stale progress screens stopped appearing after bulk imports, and the staged failover now runs as a quarterly check.

Technologies

MariaDBMaxScalePrometheusLinux
  • Support

    MariaDB Galera Quorum Loss and Split-Brain Support

    European Telecommunications OperatorTelecommunications

    Recovering a four-node Galera cluster after a site-link failure left both halves non-Primary, and redesigning it so one link fault cannot do that again.

    MariaDBGalera ClusterMaxScaleLinux
    Read case study: MariaDB Galera Quorum Loss and Split-Brain Support
  • Assessment

    MySQL Topology and Backup Recoverability Assessment

    Media Streaming ProviderSoftware

    Review of replication topology, failover readiness, backup recoverability, and configuration drift across a MySQL estate that had grown organically.

    MySQLPercona XtraBackupProxySQLTerraform
    Read case study: MySQL Topology and Backup Recoverability Assessment
  • Remediation

    MySQL Replication Lag Remediation

    E-Commerce Marketplace OperatorCross-Industry

    Resolving replica lag that grew to hours during batch windows because single-threaded apply could not keep pace with large multi-row transactions.

    MySQLProxySQLPrometheusLinux
    Read case study: MySQL Replication Lag Remediation
  • Assessment

    Cassandra Cluster Health and Capacity Assessment

    Public Sector AgencyGovernment

    Assessment covering topology, replication and consistency configuration, JVM and garbage collection behavior, compaction health, and growth headroom.

    Apache CassandraLinuxPrometheusKubernetes
    Read case study: Cassandra Cluster Health and Capacity Assessment
  • Assessment

    MongoDB Deployment Health Assessment

    Public Sector AgencyGovernment

    Structured review of schema design, index efficiency, replica set topology, and backup recoverability ahead of a major workload increase.

    MongoDBLinuxPrometheusKubernetes
    Read case study: MongoDB Deployment Health Assessment

Related Reading

Ready for a MariaDB Health Check?

AceMQ's senior MariaDB engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.