RabbitMQ HA, Disaster Recovery & Cluster Sizing: A Practical Guide

A cluster stretched across regions with real network latency runs into the Raft consensus delay covered above, and that's a real limitation. Three availability zones within a single region is a different situation. AZ-to-AZ round trips inside a region typically run one to two milliseconds, well inside what Raft consensus tolerates without treating normal latency as a failure signal. That's why one node per AZ is the standard production pattern rather than a compromise.

Three is the floor for the same reason it's the floor everywhere else in this guide, applied spatially: one node in each of three AZs gives you node-level fault tolerance and zone-level fault tolerance from the same three-node topology, with no additional nodes required to get both. A cluster running all three nodes in a single AZ has the node-level tolerance without the zone-level tolerance, which defeats the point of being in a multi-AZ region at all.

Quorum member placement follows the node placement automatically in a three-node, three-AZ cluster: each queue's three replicas land one per zone by default, since RabbitMQ places one member per node up to the configured group size. In larger clusters spanning more than three AZs or with more than three nodes per AZ, don't assume that holds; set the queue-leader-locator policy to balanced and verify with rabbitmqctl list_queues name leader members that replicas for your critical queues aren't stacking inside one zone.

pause_minority is what actually protects you when a zone goes dark, not just when a single node fails. If one AZ becomes unreachable from the other two, the isolated node pauses itself rather than risk becoming a divergent source of truth, and the two AZs that still see each other keep the cluster running as the majority. Design your alerting to expect that pause as the correct behavior for a zone outage, not as an additional incident on top of it.

Watch the latency budget as an ongoing signal, not a one-time check. If inter-AZ round trips are consistently running above roughly five milliseconds, don't tune around it as if it were a random partition; treat it as a sign the region's AZ layout or your cloud provider's network path isn't behaving like same-region latency, and investigate the topology before you investigate RabbitMQ.

When a zone does go down, expect three things to happen in order: the node in that zone drops from rabbitmqctl cluster_status on the surviving nodes, any queue that had its leader in that zone runs a new election among the surviving replicas, and every client connection that was pinned to the lost zone's node drops and needs to reconnect elsewhere. Put a load balancer or DNS entry in front of all three nodes rather than hardcoding a single node's address into client configuration, health-check each node individually so a downed zone is pulled from rotation without a manual step, and make sure client libraries have automatic connection recovery enabled so a dropped connection reconnects to a surviving node on its own instead of requiring an application restart.

A cluster that's never been broken on purpose is a cluster whose HA claims are still theoretical. Run each of these before go-live, and again after any significant topology change, with someone watching the expected observation, not just confirming the cluster came back eventually.

• Kill a node outright (kill -9 the beam process). Expect: the other two nodes report it down within one heartbeat interval, quorum queue leaders on that node re-elect, and clients with connection recovery reconnect without an application error.
• Drain a node gracefully with rabbitmqctl stop_app. Expect: leadership hands off before the process actually stops, and no unplanned election events appear in the log.
• Partition one zone from the other two at the network level. Expect: the isolated node pauses under pause_minority, the surviving majority keeps running, and no writes are accepted on both sides at once.
• Fill the disk on one node. Expect: a disk alarm fires locally, that node blocks new connections, and the rest of the cluster is unaffected.
• Exhaust memory on one node. Expect: a memory alarm fires, publishers to that node are blocked, and consumers keep draining what's already queued.
• Restart a node under sustained production-equivalent load. Expect: it rejoins and its replicas resync automatically, with no manual intervention and no visible spike in queue depth.
• Force a TLS certificate to expire (or simulate it in a lower environment). Expect: connections fail with a clear handshake error, not a silent hang, confirming your rotation runbook actually works before a real certificate lapses.
• Kill the node currently leading a critical queue specifically, not a random node. Expect: a new leader is elected within seconds and in-flight publisher confirms are retried transparently by the client library.
• Stop every node in one AZ simultaneously to simulate a full zone outage. Expect: the remaining two zones keep the cluster and every quorum queue available throughout.
• Pull one node out of load balancer rotation via its health check before touching it. Expect: traffic stops routing to it before you take any other action, with zero client-visible errors.
• Run a rolling upgrade, one node at a time. Expect: the cluster stays available for the whole sequence, and the mixed-version window closes within your planned maintenance time.
• Crash a consumer mid-acknowledgment. Expect: its unacknowledged messages return to the queue automatically once the connection closes, with no message loss.
• Inject artificial latency between two AZs rather than dropping the link entirely. Expect: this shows you what real network congestion looks like in your metrics, so you don't mistake it for a hardware failure during an actual incident.

For clusters running on managed Kubernetes, the platform-specific version of several of these drills, especially the node drain and rolling upgrade cases, is covered in RabbitMQ on EKS vs AKS vs GKE and in production RabbitMQ architecture for enterprise. If your cluster is already live and you'd rather have someone run this drill list against production-equivalent load than script it from scratch, that's the kind of engagement our Kubernetes mission-critical stabilization work and support team run regularly, backed by a 15-minute emergency response SLA if something in the drill turns up a real gap.

If the open question is the mechanism for moving messages between regions rather than how to size and place the clusters, that decision has its own guide: RabbitMQ federation vs shovel for disaster recovery.

A cluster that has never lost a node in anger will surprise you the first time it does, usually at 2am. This is the programme we run with clients before a RabbitMQ cluster carries production traffic. It takes about a day and produces numbers rather than opinions.

Pre-conditions.

• A staging cluster shaped like production: same node count, same AZ placement, same RabbitMQ version (we run this on everything from 3.8.x to 4.x), same queue types, policies, client libraries and connection settings. Quorum and classic queues fail differently, so if production has both, staging has both. See /blogs/rabbitmq-classic-vs-quorum-queues/ for why that matters here.
• Synthetic publishers and consumers with sequence numbers. Each publisher stamps messages with a stream id and an increasing counter; each consumer records every sequence it sees. That makes loss (a gap) and duplication (a repeat) countable instead of arguable. Publishers use confirms, consumers use manual acks.
• Dashboards up: queue depth, publish and deliver rates, memory and disk alarms, quorum leader count per node, connections and channels, and client-side confirm latency. If you cannot see it, you cannot pass it.
• A named runbook owner, a shared channel, and a clock.

The drills, in order.

1. Kill one quorum member. Stop RabbitMQ on a node that follows most queues. Watch leader election on the queues it led and watch confirm latency at the publishers. Pass: election completes in a few seconds, zero sequence gaps at the consumers, publisher confirms keep arriving throughout. Restart the node and confirm it rejoins and catches up.

1. Kill the node holding the most leaders. Leaders pile onto whichever node was up first, so this is the harder version of drill 1. Kill it, watch the same signals, bring it back, then rebalance:

rabbitmq-queues rebalance quorum

Pass: no loss, confirms continue, and after the rebalance the leader count is roughly even across nodes.

1. Fill a disk to the free-disk alarm. Write a large file on one node until disk_free_limit trips. Pass: publisher connections show as blocked in management, publishes stall rather than fail, consumers keep draining, and publishing resumes on its own once the file is deleted. If consumers stop too, a consumer is sharing a connection with a publisher; fix that before go-live.

1. Partition the network between nodes. With cluster_partition_handling = pause_minority, block traffic between one node and the other two using iptables. Pass: the minority node pauses itself, quorum queues on the majority side keep serving, clients on the paused node see a connection error and reconnect to a surviving host from their address list, and the cluster heals without manual intervention when the rule is removed. Record what the clients logged; that line is what your on-call engineer will see.

1. Lose an availability zone. Stop every node in one AZ at once. In a three-AZ layout a majority survives. Pass: identical to drill 1, zero loss and continuing confirms. If the load balancer takes more than a few seconds to drop the dead nodes, fix the health check now.

1. Regional failover. Stop the publishers, note the highest sequence confirmed in region A, take region A down, and fail over to region B using whichever cross-region path you chose (see /blogs/rabbitmq-federation-vs-shovel-disaster-recovery/). RPO is the count of messages confirmed in A that never appear in B; read it straight from the sequence logs. RTO is wall-clock time from the outage to consumers running against B. Pass: both numbers sit inside the targets the business signed off on. Write them down; they are the honest answer to "what is our DR posture".

1. Failback without duplicates. Bring region A back and return traffic to it. Pass: consumer sequence logs show no repeats. If they do, the consumers are not idempotent, and that is an application fix, not a broker fix.

1. Restore from backup. Export definitions with rabbitmqctl export_definitions, destroy the staging cluster, rebuild it, re-import. Pass: exchanges, queues, bindings, vhosts, users, permissions and policies all come back. Know what does not: messages. A definitions backup is topology, not data. Message survival comes from replication and the cross-region path, never from a backup file.

Cadence. Run the full programme before go-live and again after any topology change: a new node, an AZ move, a major version upgrade, a change to partition handling. Run the single-node kill quarterly; it is cheap and catches client reconnect drift nobody meant to introduce. Run the regional failover at least yearly and whenever the runbook owner changes, because a runbook nobody present has executed is a document, not a capability. Most of the failures in /blogs/top-10-rabbitmq-troubleshooting-issues-how-to-avoid-them/ would have surfaced in one of these drills first.

A sizing and DR review, drills included, is a normal AceMQ engagement; see /rabbitmq/.