BlogMariaDB

How to Run MariaDB Galera in Production: Quorum, SST, gcache

Scott Sternloff

By Scott Sternloff, Senior Enterprise Architect

LinkedIn · Updated · 5 min read

How to Run MariaDB Galera in Production: Quorum, SST, gcache
On this page

Short answer

A MariaDB Galera cluster gives you high availability only if three things are set on purpose: an odd number of voting nodes so quorum survives a failure, a gcache.size large enough that a restarted node catches up by incremental state transfer (IST) instead of a full state snapshot transfer (SST), and an alert on flow control before it throttles every writer.

A MariaDB Galera cluster gives you high availability only if three things are set on purpose: an odd number of voting nodes so quorum survives a failure, a gcache.size large enough that a restarted node catches up by incremental state transfer (IST) instead of a full state snapshot transfer (SST), and an alert on flow control before it throttles every writer.

If a cluster is already in non-Primary state or a joiner keeps repeating state transfers, our mariadb consulting team starts from these same variables.

How many nodes does a Galera cluster need to keep quorum?

Quorum means more than 50% of the last known membership can talk to each other, and that majority is called the Primary Component. Three nodes survive the loss of one. Two nodes do not: when one fails, the survivor holds exactly 50%, loses quorum and stops serving queries. A cluster with an even node count has no extra fault tolerance over the odd count below it.

If you only have two data nodes, add the Galera Arbitrator (garbd), a lightweight, stateless voting member that is not a full database node. pc.weight (default 1) lets you weight votes, which matters when one site holds several nodes, so a link failure between sites cannot leave each side thinking it has the majority.

First make sure every node lists all members in wsrep_cluster_address (gcomm://node1,node2,node3) and uses the same wsrep_cluster_name, because nodes cannot join a cluster with a different name. Then run SHOW STATUS LIKE 'wsrep%' on every node and check these variables:

Variable Healthy value Meaning
wsrep_cluster_status PRIMARY NON_PRIMARY means quorum lost, DISCONNECTED means the node has no group
wsrep_ready ON The node accepts queries
wsrep_cluster_size Your node count Nodes this node can currently see
wsrep_local_state_comment A synced state Human-readable node state; anything else means the node is joining, donating or otherwise not in sync

Failure detection is controlled by evs.suspect_timeout (PT5S) and evs.inactive_timeout (PT15S); raising them delays detection of real failures.

How do I stop a node restart turning into a full SST?

When a node rejoins, it reports the last sequence number (seqno) it applied, and a donor looks for the next seqno in its gcache. If it is there, the node catches up by IST, replaying only the missing write sets. If not, the donor sends a full snapshot (SST), which is far slower and loads the donor.

  1. Measure your write rate. Read wsrep_received_bytes twice during peak load, for example an hour apart, and divide the difference by the elapsed seconds. The MariaDB documentation uses this method.
  2. Decide how long a node may be down and still use IST. One hour of planned maintenance, or a weekend?
  3. Set gcache.size to rate times window, with margin. The default is 128M, preallocated at startup. At 20 KB per second, one hour is about 72 MB, which fits. At 5 MB per second it is 18 GB, which does not, and you need a larger gcache on every node (it is not dynamic, so it needs a restart).
  4. Put gcache.dir on fast storage. The documentation recommends SSD or NVMe.
  5. Set gcache.recover=yes explicitly. It lets a restarted node reuse its gcache and act as an IST donor right away, which helps after a whole-cluster restart. The wsrep_provider_options page lists the default as no, while the IST page calls it enabled by default in modern versions, so do not rely on the default.

Choose the SST method knowingly. rsync is the default and holds a read lock on the donor for the whole transfer. mariadb-backup is non-blocking for the donor and much faster than mysqldump, the slowest method, but the joiner must be configured very similarly to the donor (for example the same innodb_file_per_table). Set the same wsrep_sst_method on every node, because any node can end up as donor or joiner. To keep a load balancer away from a donor, wsrep_sst_donor_rejects_queries (OFF by default) makes it return an error.

What causes flow control, and how do I read it?

Galera replication is virtually synchronous: write sets are certified at commit but applied by each node afterwards. When a node's receive queue passes gcs.fc_limit (16 write sets by default), it tells the cluster to pause replication, and every writer slows to the pace of the slowest node. Replication resumes when the queue falls below gcs.fc_limit times gcs.fc_factor (1.0 by default).

Status variable What to look for
wsrep_flow_control_paused Fraction of time replication was paused since the last FLUSH STATUS; anything noticeably above 0 deserves a look
wsrep_local_recv_queue_avg Noticeably above zero means that node cannot apply write sets as fast as they arrive
wsrep_flow_control_sent Which node is sending pause requests (the slow one)
wsrep_local_send_queue_avg Above zero points to network throughput or throttling issues
wsrep_cert_deps_distance Potential parallelism, to compare with applier threads

Find the node with the high receive queue and fix that node: slow disk, a long-running backup or analytic query competing for I/O, or too few applier threads. wsrep_slave_threads defaults to 1. Raise it toward wsrep_cert_deps_distance and test, and drop it back to 1 if you see consistency problems, which is the documentation's advice. Raising gcs.fc_limit only hides the problem and increases the lag you carry.

What should the application know about Galera?

  • Only InnoDB tables are replicated. Writes to other engines are not.
  • Stale reads are possible by default. With wsrep_sync_wait=0, a read on another node may not see a write just committed. Set wsrep_sync_wait=1 for sessions that must read their own writes, at the cost of read latency.
  • Certification conflicts roll back transactions. Watch wsrep_local_cert_failures. Retry logic and short transactions reduce them, as does sending writes for one set of rows to one node.

Quorum loss is a different procedure from a slow node. The safe_to_bootstrap flag in grastate.dat decides which node can start a new Primary Component, which is the subject of our page on MariaDB Galera quorum loss support. If you want an outside review of your wsrep settings and failure drills, a mariadb consultant can run them with your team.

Sources

Frequently Asked Questions

Why does my Galera node do a full SST every time it restarts?

The donor no longer has the node's last seqno in its gcache, so IST is impossible. Size `gcache.size` from your write rate and the downtime you want to tolerate, and set `gcache.recover=yes` so the restarted node's own cache can serve others.

Can I run a two-node MariaDB Galera cluster?

You can, but it has no fault tolerance: if either node fails, the other drops below quorum. Add a third node or a Galera Arbitrator (`garbd`) so a majority survives one failure.

What does wsrep_flow_control_paused mean?

It is the fraction of time since the last `FLUSH STATUS` that replication was paused by flow control. A value near 0 is healthy. A high value means a node's receive queue keeps passing `gcs.fc_limit`, so find that node using `wsrep_local_recv_queue_avg`.

Scott Sternloff

Author

Scott SternloffSenior Enterprise Architect

Senior Enterprise Architect at AceMQ. Writes the technical coverage here: RabbitMQ, Kafka and Redis architecture, clustering and migrations.

Talk to an engineer

Need a hand with MariaDB?

Senior engineers who run it in production. Get a quote or a free consultation, no sales pitch.

Free Consultation

Get Expert Eyes on Your MariaDB Environment

Whether you're troubleshooting a production incident, planning a migration, or want a second opinion on your architecture — our team is ready. No pitch, just answers.

Email Us