The RabbitMQ Reliability Guide: Ten Failure Patterns and Their Fixes

Symptoms. The management UI reports a network partition, nodes each believe they are authoritative, and queue state diverges. On heal you see message loss, duplication, or nodes refusing to rejoin. Root cause. The Erlang distribution link drops across sites, availability zones or regions; classic mirrored queues make it worse because each side keeps accepting writes.

The fix. Set cluster_partition_handling to pause_minority in production and never ignore. Deploy an odd node count, three or five, spread across independent failure domains, keep the inter-node link on a low-latency reliable network, and favour quorum queues, which use Raft consensus and tolerate a minority failure without diverging.

Symptoms. Unexplained I/O spikes, node slowdowns and elevated disk latency, most often on Windows or hardened corporate servers, with no matching change in traffic. Root cause. Endpoint antivirus or EDR agents scan the RabbitMQ data directory in real time: the Mnesia store, the message store, the quorum-queue write-ahead log and segment files.

The fix. Exclude the node data directory, message store, quorum WAL and segment files, and logs from real-time scanning. On Linux the data directory is typically /var/lib/rabbitmq/mnesia; on Windows it sits under %APPDATA%\RabbitMQ. Confirm the real path with rabbitmq-diagnostics status before writing the exclusion. This is one of the most common mystery-latency root causes we find on-premises, and a five-minute exclusion change often recovers the lost throughput.

Symptoms. Unpredictable latency, queue stalls and occasional node restarts; CPU steal time is high and disk latency spikes under load. Root cause. Nodes share CPU, memory or disk I/O with noisy-neighbour workloads on an oversubscribed hypervisor or shared storage, and the Erlang scheduler is starved.

The fix. Pin dedicated CPU cores and reserve memory for every VM running a node. Prefer local SSD or NVMe for the data directory over shared storage. Monitor CPU steal, disk await and memory pressure rather than averages, and keep cluster nodes on separate physical hosts. We size vCPU and RAM to sustained publish and consume rates and confirm scheduler headroom under peak, not average, load.

Symptoms. Classic queues become unstable under high churn or large backlogs, failover behaviour is unpredictable, and nodes may crash under heavy load. Root cause. Classic queues, and especially legacy mirrored ones, were not built for durable high availability at scale; classic mirrored queues are deprecated.

The fix. Move durable, high-availability workloads to quorum queues. Use streams where you need high-throughput fan-out or replay. Enable publisher confirms and consumer acknowledgements, set a delivery-limit and a dead-letter exchange so poison messages do not loop, and reserve classic queues for transient, non-replicated traffic. We run classic-to-quorum migrations with zero or minimal downtime and validate that clients support publisher confirms before the cutover.

Symptoms. The unacked count in the UI climbs, memory rises, and a queue looks stuck even though consumers are connected. Redelivery storms are common. Root cause. Consumers use manual acknowledgement but do not ack, or ack slowly; unbounded prefetch lets one consumer hoard messages in flight, and a crashed consumer holds its unacked messages until the channel closes.

The fix. Set a sensible per-consumer prefetch (basic.qos prefetch_count), starting around 10 to 50 and tuning from there. Acknowledge after successful processing, preferring manual acks with confirms over autoack for durability. Set consumer_timeout to match real processing time so long jobs are not force-closed, route repeatedly redelivered messages to a dead-letter exchange with a delivery-limit, and alert on sustained unacked growth. Prefetch tuning is the single highest-leverage change we make for the queue-is-stuck-but-consumers-look-healthy ticket.

Symptoms. Connection and channel counts climb steadily, file descriptors and Erlang processes run out, and new clients cannot connect. You may see connection_forced or too_many_channels errors and what looks like a slow memory leak. Root cause. Applications open a new connection or channel per message or per request instead of reusing long-lived connections with a pool of channels; missing automatic recovery leaves orphaned connections behind.

The fix. Reuse long-lived connections and never open and close one per publish. Use one channel per thread or consumer from a bounded pool, set channel_max, monitor connection, channel and file-descriptor counts, and enable automatic connection recovery in the client libraries. Connection churn regularly masquerades as a RabbitMQ memory leak; the cause is almost always on the client side.

Symptoms. Publishers are throttled or blocked, upstream pipelines time out, and connections enter a flow state; queues grow faster than they drain. Root cause. Consumers cannot keep pace, or a memory or disk high-watermark alarm fires and RabbitMQ applies back-pressure to protect the node.

The fix. Keep queues short by matching consumer capacity to publish rate: scale consumers and use competing consumers to drain quickly. Tune vm_memory_high_watermark and disk_free_limit to the actual host and alert on alarm state. Apply max-length or TTL policies and consider on-disk behaviour for very large backlogs. Flow control is RabbitMQ protecting itself; the fix is consumer capacity and correct alarm thresholds, never disabling back-pressure.

Symptoms. Incidents surface only when customers report them; there is no baseline, and the team is blind to queue depth, memory and partition status until something breaks. Root cause. No metrics pipeline or alerting, and reliance on the management UI for a real-time-only view.

The fix. Deploy the Prometheus plugin with Grafana, or a managed equivalent. Track queue depth, unacked, publish and deliver rates, node memory and disk, file descriptors, connection and channel counts, and partition status. Alert on trends and retain history for capacity planning. We deploy a standard RabbitMQ Grafana dashboard and alert set during onboarding so problems surface before they page anyone.

Symptoms. Failover is inconsistent, synchronisation lags, and data can be lost on failover. Over-mirroring strains resources, and deprecated-feature warnings appear in the logs. Root cause. The cluster still relies on classic mirrored queues and automatic mirroring policies, or ha-mode policies are applied too broadly.

The fix. Retire classic mirroring and move high availability to quorum queues with explicit, tested policies. Define targeted policies instead of blanket mirroring, test failover regularly and monitor Raft and synchronisation state, and plan removal of any remaining mirrored-queue policies on modern RabbitMQ. Classic mirrored queues are at the end of the road; the migration playbook moves you to quorum queues with failover validated before sign-off.

Symptoms. False-positive node and connection failures, spurious partitions in cloud or WAN deployments, and connections dropped under otherwise normal load. Root cause. Default heartbeat, TCP timeout and Erlang distribution settings are not matched to higher-latency cloud or wide-area links.

The fix. Set heartbeat intervals appropriate to the environment, longer for WAN and cloud. Tune tcp_listen_options such as keepalive and buffer sizes, adjust the Erlang net_ticktime so transient latency does not trigger a partition, keep node clocks in sync with NTP, and validate under real network conditions. We calibrate net_ticktime and heartbeats to the actual link profile so ordinary cloud jitter never masquerades as a node failure.

The patterns above are failure modes; these are the routine discipline that keeps you out of them. Back up configuration and definitions, including users and vhosts, so recovery is fast and repeatable. Test at scale in a pre-production environment with real traffic and failure scenarios. Keep RabbitMQ and Erlang patched on a schedule, validated in staging first; where a version is past end of support, a CVE-patched extended build keeps you covered without a rushed upgrade. Manage HA and queue policy explicitly, with targeted, tested policies and quorum queues for durability. Document every change in an infrastructure-as-code repository for reproducibility and clean rollback.