RabbitMQ consumer_timeout: What It Does and Why Consumers Disappear

consumer_timeout is RabbitMQ's delivery acknowledgement timeout. If a consumer holds a delivery without acknowledging it for longer than the timeout, RabbitMQ closes that consumer's channel with a PRECONDITION_FAILED channel exception, and the unacknowledged messages return to the queue for redelivery. The default is 30 minutes. From RabbitMQ 4.3 it only applies to quorum queues.

It only applies to consumers in manual acknowledgement mode, the usual setting in AMQP 0-9-1 clients when messages must not be lost: the broker hands over a delivery and waits for the consumer to acknowledge messages it has finished with. The timeout exists to catch consumers that took a message and then hung, so the work is requeued rather than held forever. It also catches healthy consumers whose jobs simply take longer than 30 minutes, and that is where most of the confusion starts: the application is running, the connection is open, and the consumer count in the management UI has quietly dropped to zero.

The broker logs the channel closure with the consumer tag, channel, queue and delivery tag. The message in the RabbitMQ documentation looks like this:

Consumer 'consumer-tag-998754663370' on channel 1 and queue 'qq.1' in vhost '/' has timed out
waiting for a consumer acknowledgement of a delivery with delivery tag = 10. Timeout used: 180000 ms.

On the client side it arrives as a channel exception with PRECONDITION_FAILED. Because it closes the channel rather than the connection, the client library often stays connected and logs nothing obvious unless channel shutdown is handled. The queue now shows its messages as ready again, they get redelivered with the redelivered flag set, and the slow job may run twice.

The timeout is one of three ways a consumer stops consuming without the application asking it to. They look the same in the management UI and need different fixes:

• The acknowledgement timeout closed the channel. The log line above appears, the connection stays up, and the consumer is gone. The work took too long, or the consumer hung.
• The broker cancelled the consumer. When a queue is deleted, or the node hosting it fails, RabbitMQ sends the client an asynchronous basic.cancel. Clients advertise the consumer_cancel_notify capability, which RabbitMQ's own clients do by default, and the client library raises it through a callback such as handleCancel in the Java client. If the application ignores that callback, it keeps a channel with no consumer on it.
• The connection died. A load balancer, firewall or NAT gateway dropped an idle TCP connection, or the client process froze. Heartbeats detect this: the suggested default is 60 seconds, and after two missed heartbeats the peer is considered unreachable. TCP keepalives can do the same job at the operating-system level instead.

The order to check is broker log first, then the client's channel and cancel callbacks, then the network. The broker log tells you which of the three it was in a few seconds.

Set the timeout value globally in the RabbitMQ configuration file, rabbitmq.conf, in milliseconds:

# one hour
consumer_timeout = 3600000

Per queue, from RabbitMQ 3.12, with a policy or a queue argument. The policy is the better tool because it can be changed without redeploying the application:

rabbitmqctl set_policy queue_consumer_timeout "with_delivery_timeout\.*" '{"consumer-timeout":3600000}' --apply-to quorum_queues

The queue argument is x-consumer-timeout, set at declaration, also in milliseconds.

RabbitMQ evaluates the timeout once a minute, so values below one minute are not supported and values below five minutes are not recommended. It can be disabled in advanced.config by setting consumer_timeout to undefined, but the documentation recommends a high value, a few hours, rather than turning it off.

From RabbitMQ 4.3, delivery acknowledgement timeouts are only supported by quorum queues. Two consequences follow:

• Classic queue consumers lose the protection. A consumer that takes a message from a classic queue and hangs will hold it until the channel or connection closes. If you relied on the timeout to recover stuck classic queue consumers, you need client-side timeouts or monitoring instead.
• Moving to quorum queues can switch it on. A workload that ran happily on classic queues with hour-long jobs may start seeing PRECONDITION_FAILED channel closures after a quorum queue migration. Set a per-queue consumer-timeout for those queues as part of the migration, not after the first incident.

The wider queue-type differences are in migrating from classic queues to quorum queues.

A consumer about to hit the timeout looks healthy from the outside: connected, subscribed, with messages in flight. Three checks show it early.

Unacknowledged messages that stop moving. Compare ready, unacknowledged and the number of consumers per queue:

rabbitmqctl list_queues -p / name messages_ready messages_unacknowledged consumers consumer_utilisation

A queue whose messages_unacknowledged stays flat while messages_ready grows has consumers holding deliveries they are not finishing. consumer_utilisation well below 1.0 on a queue with a backlog says the same thing from the other side: the queue has messages but cannot deliver them, because the consumers are at their prefetch limit.

Which consumers hold them. rabbitmqctl list_consumers lists every subscription with its channel, consumer tag, whether acknowledgements are expected, and its prefetch limit. A consumer with a high prefetch on a slow queue is the first suspect.

The number of consumers over time. In Prometheus, with per-object metrics enabled so values are reported per queue rather than aggregated, alert on rabbitmq_queue_consumers dropping to zero on queues that should always be consumed, and on rabbitmq_queue_messages_unacked rising without a matching rise in delivered and acknowledged rates. Those two alerts catch the timeout, a broker-initiated cancel and a dead connection, whichever happens first.

Raising the timeout is the right fix only when the work genuinely takes that long. The rest is client behaviour:

• Size the timeout per queue. Set consumer-timeout on the queues that carry long jobs, a margin above the longest legitimate processing time, and leave the global default alone.
• Keep prefetch low for slow work. Every message prefetched is already a delivery the timeout is counting. With a prefetch of 50 and jobs that take a minute each, the 31st delivery has been held for more than 30 minutes before the consumer even starts it, and the channel is closed at the default timeout. See RabbitMQ prefetch count.
• Acknowledge on the channel that received the delivery, and on every code path, including the exception path. A missing ack on an error branch is the most common cause of a consumer that holds deliveries until it times out.
• Handle channel shutdown and cancel notifications. On a channel closure or a basic.cancel, open a new channel and subscribe again. Automatic connection recovery in client libraries reconnects dropped connections; it does not always cover a single closed channel or a broker-initiated cancel.
• Make processing idempotent. A timed-out delivery is requeued and redelivered, so a job that was 90 percent done runs again. Messages that can never finish in time belong in a dead letter queue rather than a redelivery loop; see RabbitMQ dead letter queues.
• Alert on unacknowledged messages and consumer count per queue, not just on queue depth. A consumer count of zero on a queue that should always have consumers is the earliest signal of all three failure modes.

These patterns show up regularly in the RabbitMQ support tickets we handle: consumers reported as missing while the application is healthy, after a quorum queue migration or a change to memory limits. If you need a second pair of eyes on a cluster doing this, AceMQ RabbitMQ support covers it, and the RabbitMQ troubleshooting guide covers the wider diagnostic path.

• RabbitMQ — Consumers: delivery acknowledgement timeout
• RabbitMQ — Consumer cancel notification
• RabbitMQ — Detecting dead TCP connections with heartbeats and TCP keepalives