RabbitMQ outage runbook

Commands here were checked against the RabbitMQ documentation (partitions guide, alarms guide, quorum queues guide and the CLI manuals) on 26 October 2026. They assume a 4.x cluster: Khepri is the default metadata store from 4.2.0, and from 4.3.0 Mnesia and the partition-handling strategies are gone. Classic mirrored queues were removed in 4.0, so this runbook covers quorum queues and streams only. Run every command against the node you are diagnosing, or add -n rabbit@HOSTNAME. Replace names in capitals.

Run these in order and write down the output. Each line answers one question.

# Is the node up, and can the CLI talk to it?
rabbitmq-diagnostics ping
rabbitmq-diagnostics erlang_cookie_sources     # only if the CLI cannot connect
# Is the RabbitMQ application running on it?
rabbitmq-diagnostics check_running
# Who is in the cluster right now?
rabbitmq-diagnostics cluster_status
# Is a memory or disk alarm in effect anywhere?
rabbitmq-diagnostics alarms
# Would stopping this node cost a queue its majority?
rabbitmq-diagnostics check_if_node_is_quorum_critical
# Where are the messages piling up?
rabbitmqctl list_queues -p / name type messages_ready messages_unacknowledged consumers
# Save the evidence before you touch anything
rabbitmq-diagnostics log_tail --number 200

Then match the symptom: publishers hang or connections show as blocked means an alarm (steps 2 and 3); a node missing from the running nodes list means a partition or a dead node (step 4); messages ready growing with consumers at 0 means a backlog (step 6); clients refused or timing out while the node looks healthy means a connection problem (step 7). Do not restart nodes yet. A restart clears the alarm's evidence, and on a cluster that has lost its majority it can leave you with fewer running nodes, not more.

From our engagements. AceMQ handles 40+ Sev 1 incidents a year for 160+ customers. In the 145 RabbitMQ support tickets we analysed, 57 (39%) described service impact in the customer's own words, and no single category dominated: queues, consumers and message flow was the largest at 24 tickets (17%), cluster health, partitions and node recovery came next at 22 (15%). Another 23% sat on the client side, in message flow, load balancers and connection settings, where the broker was usually doing what it was told. Check the clients (step 7) before deciding the broker is at fault.

When free space on the node's data partition drops below disk_free_limit (50 MB by default), RabbitMQ blocks publishing connections. The alarm is cluster-wide: one node under the limit blocks publishers on every node, while consumers keep working. Applications see stalled publishes, not errors.

rabbitmq-diagnostics alarms
rabbitmq-diagnostics check_local_alarms -n rabbit@NODE
rabbitmqctl list_connections name state          # blocked or blocking
df -h /path/to/data/directory                    # OS command, on the node
rabbitmqctl list_queues -p / name type messages message_bytes consumers

The usual cause is a backlog: a queue with a large message_bytes and no consumers. Other causes are logs, erl_crash.dump files and anything else on the same volume. To free space, restart or scale the consumers so the backlog drains, extend the volume, or remove non-RabbitMQ files. Never delete files from the node's data directory by hand. rabbitmqctl purge_queue -p / QUEUE_NAME removes every ready message and cannot be undone, so get a business sign-off first.

Lowering the limit with rabbitmqctl set_disk_free_limit VALUE clears the alarm but leaves the node free to fill the disk, and it lasts only until the next restart. The documentation suggests a conservative limit, about the amount of RAM on the node, because messages paged out under memory pressure can fill a small margin between disk checks. Confirm recovery:

rabbitmq-diagnostics check_alarms        # exits non-zero while any alarm is in effect
rabbitmqctl list_connections name state

By default a node raises the alarm at 60% of detected RAM (vm_memory_high_watermark.relative = 0.6) and blocks publishing connections cluster-wide. Service resumes by itself once consumers deliver and acknowledge enough messages for usage to fall back. The limit is a throttle point, not a cap: a node can use more memory than it.

rabbitmq-diagnostics alarms
rabbitmq-diagnostics memory_breakdown --unit gigabytes
rabbitmqctl list_queues -p / name type messages_ready messages_unacknowledged consumers
rabbitmqctl list_connections name state

Read the breakdown by category. Large quorum_queue_procs or queue_procs points at queues, binary at message bodies and properties held in memory, connection_readers at too many connections or channels, mgmt_db at the management statistics database, and allocated_unused at memory the runtime holds but is not using. Each has a different fix: get consumers back, cap prefetch, close leaked connections, or stop publishing large messages into a backlog.

A stopgap that survives only until the node stops: rabbitmqctl set_vm_memory_high_watermark 0.7, or in containers an absolute value such as rabbitmqctl set_vm_memory_high_watermark absolute "4G", which the documentation prefers on Kubernetes. To stop all publishing on purpose while you work, set it to 0, then set the previous value back. Do not restart the node to clear the alarm: you lose the evidence and push its load onto the others.

On 4.3 and later there is one metadata store, Khepri, and it is Raft-based like quorum queues and streams. There are no partition-handling strategies to configure, and no state where both sides of a partition accept conflicting writes. A side that cannot reach a majority stops making progress until it can. Three nodes tolerate the loss of one, five tolerate two, and two data centres give no protection against losing the larger site: the documentation calls three data centres the practical minimum. On 4.2 or earlier with Mnesia the recovery rules differ, so use the guide for your version.

rabbitmq-diagnostics cluster_status                  # an unreachable node is missing from running nodes
rabbitmq-queues quorum_status --vhost "/" "QUEUE_NAME"   # leader and followers of one quorum queue
rabbitmq-streams stream_status STREAM_NAME            # leader and replicas of a stream
rabbitmq-diagnostics check_if_node_is_quorum_critical

If the leader is cut off, the majority side elects a new one, usually in a few seconds. Writes are rejected meanwhile, publisher confirms can arrive late or as a negative acknowledgement, and unacknowledged deliveries are requeued, so consumers may see a message twice. If a follower is cut off, the majority side carries on; the minority side serves cached metadata reads and quorum queue deliveries pause.

What to do: fix the network or the host, not RabbitMQ. Do not reset or delete a partitioned node's data. When connectivity returns, members catch up from the leader's log and become eligible leaders again once they have caught up. Re-run cluster_status and quorum_status to confirm.

If a node is gone for good, remove it from a running node so the cluster stops waiting for it, which also removes its quorum queue and stream replicas:

rabbitmqctl forget_cluster_node -n rabbit@SURVIVOR rabbit@LOST_NODE

If most of a quorum queue's members are permanently lost, the documentation says the queue is unavailable and has to be force deleted and recreated, and the messages in it are gone. That is the moment to call an engineer. A node lost to a patch is usually avoidable: before stopping the next node in a rolling restart, run rabbitmq-diagnostics check_if_node_is_quorum_critical (a non-zero exit means stopping it would drop a queue below its majority), and use rabbitmq-upgrade drain and rabbitmq-upgrade revive to put a node into maintenance mode and bring it back.

1. Protect the data. Do not delete data directories or start empty nodes that would form a new cluster.
2. Start the nodes. A node restarted after a full shutdown waits for its peers. The last node to go down should be the first back up; after a power loss every node may believe it was not last. On the node you are sure stopped last, rabbitmqctl force_boot (run while the node is down) tells it to start unconditionally, and changes made to the cluster after that node stopped are lost. If you cannot tell which node stopped last, stop here and call.
3. Wait for a majority. Queues and metadata resume only when most replicas are online: two of three nodes, three of five.
4. Verify. Run the triage commands again, then check_if_node_is_quorum_critical on each node before anyone stops one.
5. Reconnect clients. Publishers that use confirms republish what was not confirmed; consumers will see redeliveries, so processing must be idempotent.

rabbitmqctl list_queues -p / name type messages_ready messages_unacknowledged consumers
rabbitmqctl list_consumers -p /
rabbitmqctl list_connections name state

messages_ready high with consumers at 0 means the consumers are down or cannot connect: restart them or scale them out first. messages_unacknowledged high means consumers hold deliveries and are slow or stuck. A delivery that is not acknowledged within the delivery acknowledgement timeout, 30 minutes by default, closes the channel and requeues the messages. On quorum queues from 4.0 a message redelivered more than the default delivery limit of 20 is dropped, or dead-lettered if a dead-letter exchange is set, so check the dead-letter queue for a poison message before blaming throughput.

Order of action: restore consumers, find and isolate any poison message, then scale. Raise prefetch only if consumer utilisation is low. Purge only with sign-off. A backlog that has grown for hours also raises the disk and memory alarms above, so watch both while it drains.

A RabbitMQ node can be healthy and still unreachable, and client connection or connectivity issues look like an outage to client applications. RabbitMQ's networking troubleshooting guide works in this order: check the client configuration, the server configuration, the server logs, hostname resolution, port usage, IP routing and, only if needed, a traffic capture, then authentication. Stop at the first check that fails.

# Which ports is the node listening on? A port missing here cannot accept connections
rabbitmq-diagnostics listeners
# Is any OS process listening on the AMQP port?
sudo ss --tcp -f inet --listening --numeric --processes
# From the client host: does the port accept a TCP connection?
telnet BROKER_HOST 5672
# What do current connections look like?
rabbitmqctl list_connections name state
sudo netstat --all --numeric --tcp --programs

Know the ports. AMQP 0-9-1 and AMQP 1.0 clients use 5672, or 5671 with TLS. The RabbitMQ management plugin serves the HTTP API on 15672 (15671 with TLS), the Prometheus plugin serves its metric endpoint on 15692 (15691 with TLS), and inter-node and CLI traffic uses 25672, with epmd on 4369. If the management API or the Prometheus metric endpoint answers while 5672 does not, the node is up and the fault is the AMQP listener, a firewall or a load balancer rule.

Read the telnet result. A connection that the server then closes after invalid input is healthy. A refusal by the OS means no process is listening on that port. A timeout points at a firewall, proxy or load balancer, or at a non-standard port. ping and traceroute use ICMP while AMQP uses TCP, so a successful ping proves nothing. A client without TLS on a TLS port connects and then times out, and a TLS client on a plain port makes the server log a protocol parser exception, so check the server logs with rabbitmq-diagnostics log_tail.

Inspect the log files and the TCP connections too. Many connections in TIME_WAIT point at connection churn, with clients opening and closing a connection per message. A node that has exhausted its file descriptors or other resources cannot accept new connections, so compare the open-file limit with connection counts before you blame the network. If the checks above pass and clients still fail, capture traffic with tcpdump or Wireshark, and last verify that the client can authenticate to the virtual hosts it asks for. Commands and port numbers are from the RabbitMQ networking troubleshooting guide and networking guide, read on 26 October 2026.

1. Keep the evidence: logs, cluster_status, alarms and memory_breakdown output, and the queue depths from the time of the incident.
2. Find the root cause before the next peak: why consumers stopped, why a node left, why the volume filled.
3. Set limits that fit the node: an absolute memory watermark and a disk limit sized to RAM, in the configuration file and not only at runtime.
4. Alert early. A simple check for monitoring is rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms, plus queue depth and consumer count from Prometheus or the management UI.
5. Give every quorum queue a dead-letter route so redelivery limits do not discard messages silently.
6. Get a written review. AceMQ writes a root-cause analysis after every P1; a health check finds the next failure before it happens.