Back to all use cases

RemediationSoftware / Digital Platforms

Making a node restart cost minutes instead of an afternoon

Regional Logistics Software Provider

Stopping a three-node Galera cluster from running a full state transfer every time a node restarted, by sizing the gcache to the maintenance window.

Overview

A logistics software provider ran its order and tracking database on a three-node MariaDB Galera cluster. Every kernel patch cycle meant restarting each node, and every restart ended in a full state transfer that desynced a donor for hours. Patching had become a weekend project with a real risk of a second node dropping out.

Challenge

The gcache was still at its default size, far smaller than the writesets the cluster produced during a normal maintenance window. A returning node therefore could not get an incremental state transfer and was forced into a full SST, copying the whole data directory from a donor that stopped serving writes at full speed while it did so. Flow control then slowed the application for the duration.

Environment

Three-node MariaDB 10.6 Galera cluster on virtual machines in the customer's own data centre, around 1.2TB of data, fronted by HAProxy.

Approach

AceMQ measured the cluster's real writeset rate across a week instead of sizing from a rule of thumb, then worked out how long a node is realistically away during a patch. The fix was sized against the longest window, with the SST method and donor selection reviewed for the cases where a full transfer is still unavoidable.

Solution

  1. Measured writeset volume per hour from wsrep status over a full week, including the nightly batch
  2. Resized gcache.size to cover the longest observed maintenance window with headroom, and confirmed disk space for it
  3. Switched the SST method to mariadb-backup and set a dedicated donor so a full transfer no longer desynced the busiest node
  4. Checked ports 4444, 4567 and 4568, credentials and free disk on both sides of the transfer path
  5. Rewrote the patching runbook to restart one node at a time and wait for it to report Synced before the next
  6. Added alerts on wsrep_local_state_comment, flow control pause and gcache coverage in minutes

Outcome

The next three patch cycles rejoined every node by incremental state transfer in minutes, with no full SST and no application slowdown. The customer now schedules patching in business hours and owns a runbook that states how long a node can stay away.

Technologies

MariaDBGalera ClusterPrometheusLinux
  • Support

    MariaDB Galera Quorum Loss and Split-Brain Support

    European Telecommunications OperatorTelecommunications

    Recovering a four-node Galera cluster after a site-link failure left both halves non-Primary, and redesigning it so one link fault cannot do that again.

    MariaDBGalera ClusterMaxScaleLinux
    Read case study: MariaDB Galera Quorum Loss and Split-Brain Support
  • Assessment

    MariaDB Replication and MaxScale Failover Assessment

    Online Learning PlatformEducation

    Reviewing a replicated MariaDB estate behind MaxScale and finding stale reads, a failover script written for MySQL GTIDs, and tables with no primary key.

    MariaDBMaxScalePrometheusLinux
    Read case study: MariaDB Replication and MaxScale Failover Assessment
  • Remediation

    MySQL Replication Lag Remediation

    E-Commerce Marketplace OperatorCross-Industry

    Resolving replica lag that grew to hours during batch windows because single-threaded apply could not keep pace with large multi-row transactions.

    MySQLProxySQLPrometheusLinux
    Read case study: MySQL Replication Lag Remediation
  • Consulting

    MariaDB 10.6 to 11.8 Upgrade Planning on Galera

    Healthcare Benefits AdministratorHealthcare

    Planning and rehearsing a rolling upgrade from MariaDB 10.6 to 11.8 on a Galera cluster, around the optimizer, TLS and change buffer differences.

    MariaDBGalera ClusterTerraformLinux
    Read case study: MariaDB 10.6 to 11.8 Upgrade Planning on Galera

Related Reading

Facing a MariaDB Production Issue?

AceMQ's senior MariaDB engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.