Overview
A telecom operator ran subscriber provisioning on a MariaDB Galera cluster split across two sites, two nodes each. When the link between the sites dropped for several minutes, neither side held a majority and both stopped accepting writes. Provisioning stopped, and in the confusion an engineer bootstrapped both halves separately.
Challenge
Two sites with two nodes each can never keep quorum when the link between them fails, because neither side has more than half the votes. The emergency bootstrap of both halves then produced two diverged copies of the data. The team had to decide which copy was authoritative while provisioning requests queued upstream.
Environment
Four-node MariaDB 10.11 Galera cluster over two sites, MaxScale for routing, a mix of virtual machines and physical servers.
Approach
AceMQ joined the incident bridge under the 15-minute SLA and compared the committed sequence number on every node before touching anything else. Once the authoritative copy was known, recovery was a controlled rebuild, followed by a topology change that removed the even-node weakness.
Solution
- Read seqno and safe_to_bootstrap from grastate.dat on every node to identify the most advanced copy
- Bootstrapped one node from that copy and rebuilt the other three by state transfer rather than merging diverged data
- Extracted the rows written only on the discarded side during the split and replayed the valid ones through the application
- Added a Galera arbitrator at a third location so a single site-link failure leaves a majority
- Set pc.weight on the nodes to reflect the primary site and documented the order of recovery
- Wrote a quorum-loss runbook with the exact checks to run before anyone bootstraps a node
Outcome
Provisioning resumed the same day on a single authoritative copy, with the small set of split-period changes replayed by hand. A later inter-site link fault left the cluster serving writes on the majority side, and nobody had to bootstrap anything.
Technologies
Linux
Kafka
PostgreSQL
