RabbitMQ Incident Response: The First 15 Minutes

The first fifteen minutes of a RabbitMQ incident decide whether the next hour is spent fixing the broker or discovering that the broker was never the problem. The order below is the one an on-call engineer who has run RabbitMQ in production follows: classify first, act second, and hand over evidence rather than guesses if the incident outgrows the team.

It assumes you can reach a cluster node, or the management interface, with an account that can run rabbitmqctl and rabbitmq-diagnostics. If you cannot, that is the first finding.

Minutes 0 to 2: is RabbitMQ the cause or the casualty?

Most pages that say RabbitMQ is down describe an application that stopped receiving messages. That has at least four causes, and only one of them is the broker. Settle it before touching anything.

• Is a node answering? rabbitmq-diagnostics ping against each node, or GET /api/overview on the management port. A node that answers is degraded, not down, and the two are handled differently.
• Is the application connected? rabbitmqctl list_connections name state. Zero connections from a service that should have dozens is a client-side or network failure, not a broker failure.
• What changed? A deploy, a config push, a certificate rotation, a node reboot, a traffic spike. Incidents that follow a change are usually the change.

Minutes 2 to 5: read the alarms before the logs

RabbitMQ has two resource alarms and both produce the same symptom, which is why teams misread them: publishers block, consumers keep working, and nothing crashes. The application sees publishes hang, queue depth stops rising, and every graph looks calm.

• Memory alarm. Fires when a node passes the memory high watermark, 40% of RAM by default on 3.x and 60% on 4.x. rabbitmq-diagnostics alarms lists it; rabbitmq-diagnostics memory_breakdown tells you whether the memory is queue contents, binaries, connections or metadata.
• Disk alarm. Fires when free space on the data directory falls below disk_free_limit. Message stores, quorum queue logs and the node's own log files compete for the same disk.
• Telling an alarm from an outage. Connections in list_connections show blocked or blocking. The alarm is cluster-wide: a memory alarm on one node blocks publishers on every node.

Clearing an alarm is the one thing you can safely do in the first five minutes. Consume the backlog, add disk, or raise the watermark temporarily on the affected node. A restart is not required and usually makes it worse by replaying the same load into a fresh node.

Minutes 5 to 8: cluster and quorum state

rabbitmqctl cluster_status answers which nodes the cluster expects and which are reachable. What a partition means depends on the version, and the difference matters in the first minutes.

• 3.x and 4.0 to 4.2: a recorded partition. The output has a partitions section. With cluster_partition_handling = pause_minority, the minority side pauses itself and its queues are unavailable until the network heals. That is the setting working, not a second failure, and restarting the paused node does not shorten it.
• 4.3 and later: no partition strategy at all. Mnesia and the partition-handling modes are gone; the metadata store, quorum queues and streams are all Raft, so the only question is whether each still has an online majority. rabbitmq-diagnostics check_if_node_is_quorum_critical answers it for the node in front of you.
• A node down with no partition. Quorum queues with a surviving majority keep serving. A classic queue that lived on the lost node is unavailable until it returns; on 3.x mirroring could cover that, and 4.0 removed mirroring. This is the largest single argument for quorum queues, and it is usually discovered here.
• Quorum lost. rabbitmq-queues quorum_status <queue> shows the members and the leader. A quorum queue without a majority holds publishes and acknowledgements until enough members return, or they time out. Do not delete and recreate it under pressure; the data is on the surviving members.

Minutes 8 to 12: queues, consumers and the unacknowledged count

One command covers most of it:

rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers

The three numbers together tell you which of four situations you are in:

• Ready climbing, consumers zero. The consumer fleet is down or disconnected. The broker is doing its job; the fix is on the consuming side.
• Ready climbing, consumers present, unacknowledged high. Consumers are receiving and not acknowledging. Each holds up to its prefetch count in flight, so a stalled consumer with a large prefetch pins messages nobody is processing. See prefetch count for the sizing, and look for a downstream dependency timing out inside the handler.
• Ready climbing, consumers present, unacknowledged near zero. Consumers are simply slower than producers. Scale consumers or throttle producers; nothing on the broker is wrong.
• Everything flat, publishes failing. Go back to the alarms; you skipped one.

While you are there, rabbitmqctl list_channels and a second look at connections will show churn: connection counts that rise and fall every few seconds mean a client is reconnecting in a loop, which is a symptom and, on a large enough fleet, a cause.

Minutes 12 to 15: name the severity and stop the bleeding

By now the incident has a shape. Give it a severity so the right people are woken and the wrong actions are not taken.

• Sev 1: publishers blocked or the cluster unreachable. Resource alarm, lost quorum, or every node down. Business impact is immediate. This is the case for outside help if the team does not run RabbitMQ daily.
• Sev 2: degraded. One node down with quorum intact, a minority paused by pause_minority on a pre-4.3 cluster, replication lag. Traffic is flowing; capacity or redundancy is reduced.
• Sev 3: client-side. Consumers down, a reconnect loop, a slow handler. The broker is healthy and the fix belongs to the application team.

Actions that are safe under pressure: consuming or draining a backlog, adding disk, temporarily raising a watermark, scaling consumers, and restarting an application. Actions that are not: purging a queue, which is data loss; force_boot or forcibly removing a node, which can discard the only good copy of a queue; and restarting nodes during a partition, which extends it.

What to have ready before you call anyone

If the incident goes to a support provider, the first request is the same every time, and it takes ten minutes to assemble under pressure or one minute if it is already in hand:

• rabbitmqctl version and the Erlang version, per node.
• The output of rabbitmqctl cluster_status, rabbitmq-diagnostics status and rabbitmq-diagnostics alarms.
• A definitions export from rabbitmqctl export_definitions, which captures every queue, exchange, binding, policy and user without the messages.
• The last hour of node logs, from the RabbitMQ log directory or the journal.
• The change that preceded the incident, and the time it landed.

What a fifteen-minute response commitment does and does not mean

A response commitment is a promise that a named engineer makes contact and begins work within the window. It is not a resolution time, and a provider that promises resolution for a problem it has not diagnosed is promising something it cannot know. What matters in the contract is who responds, how quickly the severity is confirmed, and how escalation works if the first engineer cannot close it. That is the shape of AceMQ's RabbitMQ support, which carries a fifteen-minute emergency response across 3.8 through 4.x, and it is why the evidence list above is an onboarding checklist rather than something assembled on the night.

When the incident is over, the useful question is which of the fifteen minutes above took longest to answer, because that is where the next one will be lost. A health check finds the alarm thresholds, queue types and partition settings that turned a routine event into an incident; RabbitMQ troubleshooting covers the cases that need an engineer on the cluster rather than a checklist.

Making the next incident shorter

• Alert on the alarms themselves, not on their symptoms. The monitoring and alerting guide covers what to page on.
• Move the workloads that matter to quorum queues, so a node loss is a Sev 2 rather than a Sev 1.
• Keep a current definitions export somewhere that is not the cluster. The disaster recovery guide sets out what else belongs beside it.
• Read the ten failure patterns before one of them reads you; most incidents are one of them.
• Write down who may authorise a restart, a failover or a purge, so the decision is not made at 3am by whoever is awake.