RabbitMQ Dead Letter Queues: The Enterprise Guide

Dead-lettering is how RabbitMQ moves a message it can no longer deliver to somewhere you can deal with it. Configured well, it turns a bad payload into a ticket. Configured badly, it is the most common way we see production messages disappear without a trace. This guide covers the mechanism as it behaves on RabbitMQ 3.13 and 4.x, the retry patterns built on it, and the operational habits that keep a dead letter queue honest.

What dead-lettering is, precisely

A queue dead-letters a message for exactly four reasons:

1. A consumer rejected it with basic.reject or basic.nack and requeue=false.
2. Its per-message TTL, or the queue's message-ttl, expired.
3. The queue was over its max-length or max-length-bytes limit and dropped the message from the head. This is the default drop-head overflow behaviour; with reject-publish the broker refuses the new publish instead and nothing is dead-lettered.
4. On a quorum queue, the message was returned to the queue more times than its delivery-limit allows.

The broker records these as rejected, expired, maxlen and delivery_limit. Nothing else counts. A queue that expires via x-expires is deleted with its contents. A purge discards messages. A message the broker cannot route at publish time is not dead-lettered either; that is the job of the mandatory flag or an alternate exchange.

A dead letter exchange (DLX) is an ordinary exchange. There is no special type and no flag on the declaration. It is simply the exchange a queue republishes to when it dead-letters, using the queue's dead-letter-routing-key if one is set and the message's original routing key otherwise. Because it is ordinary, you can fan out to a parking-lot queue and an audit queue, route by original key through a topic exchange, or send each team's failures to their own queue. A "dead letter queue" is just whatever queue you bound to it.

Two things change on the message as it is republished. The expiration property is cleared, so a message that expired once does not immediately expire again; the original value is preserved in x-death as original-expiration. And the x-death header is added or updated. Body, message id, correlation id and every other property survive intact.

Configuring the dead letter exchange: arguments versus policies

You can attach a DLX to a queue two ways. Queue arguments are set by the client at declaration time:

{
  "x-queue-type": "quorum",
  "x-dead-letter-exchange": "orders.retry",
  "x-dead-letter-routing-key": "order.failed"
}

Policies set the same behaviour from the broker side with the keys dead-letter-exchange and dead-letter-routing-key, matched to queues by a name pattern.

Prefer policies. Queue arguments are immutable: to change a DLX set as an argument you delete the queue and redeclare it, which means draining it first and coordinating every client that declares it. Policies change at runtime, apply to every matching queue, live in definitions exports, and sit in one place an operator can read. When both are present, the argument wins, which is the second reason to avoid arguments: an application that sets x-dead-letter-exchange pins a value operations cannot override without a deploy. The exception is x-queue-type, which must be an argument, and that is fine because a queue's type never changes.

One trap: only one policy applies to a queue, the highest-priority match. If you define one policy for dead-lettering and another for max-length or delivery-limit on the same queues, only one takes effect. Put every key a queue needs in the same policy definition.

When the broker dead-letters a message it appends to the x-death header, an array with one entry per queue-and-reason pair. If the same queue dead-letters the same message for the same reason again, that entry's count increments and it moves to the front of the array. Each entry records the queue, the reason, the exchange and routing keys the message carried, a timestamp, and original-expiration when TTL was involved. The broker also sets x-first-death-reason, x-first-death-queue and x-first-death-exchange once, and keeps the x-last-death-* equivalents current. An illustrative header after three laps of a retry loop:

x-death:
  - queue: orders.work
    reason: rejected
    exchange: orders
    routing-keys: [order.created]
    count: 3
    time: 1788430492
  - queue: orders.wait.30s
    reason: expired
    exchange: orders.retry
    routing-keys: [order.created]
    count: 3
    time: 1788430462
    original-expiration: "30000"
x-first-death-reason: rejected
x-first-death-queue: orders.work
x-first-death-exchange: orders
x-last-death-reason: rejected
x-last-death-queue: orders.work
x-last-death-exchange: orders

Dead-lettering can form a cycle, and the broker has one rule for it: if a message reaches the same queue twice and no hop in the cycle was a rejection, the message is dropped. If any hop was a consumer rejection, the cycle continues. So retry loops driven by requeue=false work, because every lap includes a rejection, while a loop built purely from TTL, such as a parking lot with a message-ttl that dead-letters back to the work queue, silently drops messages on their second lap.

Quorum queues change the rules

Quorum queues count redeliveries in the x-delivery-count header, and the delivery-limit policy key (or x-delivery-limit argument) caps it. Past the cap, the queue dead-letters the message if a DLX is configured and drops it if not. Starting with RabbitMQ 4.0 the default delivery limit is 20; before that it was unlimited.

That default is what surprises teams migrating from classic queues. A consumer that catches an exception and calls basic.nack with requeue=true used to spin forever on a classic queue, wasting CPU but losing nothing. On a 4.x quorum queue with no DLX, the same code drops the message after 20 attempts, and the only trace is a counter. Set delivery-limit to a value you chose and always pair it with a DLX.

Quorum queues also support two dead-letter strategies via the dead-letter-strategy policy key. The default, at-most-once, republishes without confirmation. If the DLX does not exist, or the target queue is unavailable or refuses the message, it is gone. This is the guarantee classic queues have always had.

at-least-once keeps the dead-lettered message in the source queue until every queue bound on the DLX has confirmed it. It requires a quorum queue source, overflow set to reject-publish, and a DLX. Internally the queue leader runs a dead-letter worker, which the documentation calls the dead-letter consumer: it reads dead-lettered messages from the source queue, publishes them to the DLX with confirms, and retries anything unconfirmed. Three consequences follow. If the DLX is missing or the target is down, messages accumulate in the source queue, counting against its length and disk, by design. Because delivery is at-least-once, the DLQ can receive duplicates. And the internal dead-letter consumer is plumbing, not your application; you still need a real consumer on the dead letter queue.

Retry patterns that do not lose messages

The basic retry loop uses a wait queue. The work queue's DLX is a retry exchange, which routes into a wait queue with a queue-level message-ttl and no consumers. The wait queue's DLX is the work exchange, with no dead-letter-routing-key so the message re-enters under its original routing key. A consumer that hits a transient failure rejects with requeue=false; the message waits out the TTL and comes back. The consumer reads the attempt count from the x-death entry for the work queue with reason rejected, and at the cap moves the message to a parking lot instead of rejecting again.

For exponential backoff, use one wait queue per delay tier: wait.10s, wait.1m, wait.10m. Do not do this with per-message TTLs in one queue. Expired messages are only removed from the head of a queue, in both classic and quorum queues, so a ten-minute message at the head blocks a ten-second message behind it. With tiered queues the consumer picks the tier from the attempt count, publishes to the retry exchange with the tier's routing key, waits for the publisher confirm, and only then acks the original. Acking before the confirm arrives is where consumer-driven retries lose messages. Keep the DLX on the work queue anyway as the safety net for delivery-limit.

The delayed message exchange plugin is the alternative: publish with an x-delay header and the exchange holds the message until the delay elapses. The trade-off is durability. Delayed messages live in a local store on the node hosting the exchange and are not replicated, so losing that node loses every pending message, and a large backlog of pending messages degrades the plugin. There is also no queue to inspect. For a handful of short delays it is convenient; for a retry backlog you might need to examine at 3am, wait queues are boring, visible, and replicated when declared as quorum queues.

Retries only help with transient failures. A message that fails deterministically, because the payload is malformed or references a record that does not exist, should go straight to the parking lot, so distinguish retryable from non-retryable exceptions in the consumer. And every mechanism on this page can deliver the same message twice: retries, replays and at-least-once dead-lettering guarantee delivery, not uniqueness. Consumers must be idempotent, keyed on the message id or a business key, or a retry turns a failure into a duplicate charge.

Worked example: order processing with a wait queue and a parking lot

The topology:

• Exchange orders (topic) with quorum queue orders.work bound on order.*.
• Exchange orders.retry (fanout) with quorum queue orders.wait.30s, 30 second TTL, dead-lettering back to orders.
• Exchange orders.dlx (fanout) with quorum queue orders.parking, no TTL, no automation.

Two policies cover it:

rabbitmqctl set_policy --vhost orders orders-work '^orders\.work$' \
  '{"dead-letter-exchange":"orders.retry","dead-letter-strategy":"at-least-once","overflow":"reject-publish","delivery-limit":5}' \
  --apply-to queues --priority 10

rabbitmqctl set_policy --vhost orders orders-wait '^orders\.wait\.' \
  '{"message-ttl":30000,"dead-letter-exchange":"orders","dead-letter-strategy":"at-least-once","overflow":"reject-publish"}' \
  --apply-to queues --priority 10

The consumer enforces the attempt cap and parks explicitly, publishing with a confirm before acking:

MAX_ATTEMPTS = 3

def on_message(msg):
    attempts = death_count(msg.headers, queue="orders.work", reason="rejected")
    try:
        process_order(msg)                 # idempotent on msg.message_id
        msg.ack()
    except RetryableError as err:
        if attempts + 1 >= MAX_ATTEMPTS:
            park(msg, reason=str(err))
        else:
            log.warning("retry", id=msg.message_id, attempt=attempts + 1, err=err)
            msg.reject(requeue=False)      # -> orders.retry -> orders.wait.30s -> orders
    except NonRetryableError as err:
        park(msg, reason=str(err))

def park(msg, reason):
    headers = dict(msg.headers, **{"x-parked-reason": reason, "x-parked-at": now_iso()})
    publish_with_confirm("orders.dlx", msg.routing_key, msg.body, msg.properties, headers)
    msg.ack()                              # ack only after the broker confirmed the park

Each piece protects against something specific. A crash mid-processing requeues the message; five in a row trips delivery-limit and the message goes to the wait queue rather than being dropped. A transient failure waits 30 seconds and returns, up to three attempts. A permanent failure is parked on first sight. At-least-once dead-lettering means the retry hop cannot lose a message if orders.wait.30s is briefly unavailable.

Operating a dead letter queue

A dead letter queue is only useful if someone knows when it fills. Four signals cover most of it.

Depth: rabbitmq_queue_messages for the parking lot and each wait queue, from the Prometheus plugin with per-object metrics or the detailed endpoint's queue_coarse_metrics family. Rate, which detects change sooner than depth: the rabbitmq_global_messages_dead_lettered_*_total counters break dead-lettering down by reason and, for at-least-once, by confirmation. Consumer counts, which tell you the topology is intact: rabbitmq_queue_consumers should be non-zero on the parking lot and exactly zero on every wait queue, because a consumer on a wait queue defeats the delay. Age, which RabbitMQ does not expose directly; approximate it from the x-parked-at header or the time in x-death when you sample the queue.

Starting thresholds, tuned to the service:

• Any dead-lettering on a queue that normally sees none: warn immediately, the first one is the cheapest to investigate.
• Dead-letter rate above one percent of publish rate over five minutes: page.
• delivery_limit counter increasing at all: investigate, this means crashing consumers or requeue loops, not business failures.
• Parking lot depth above a count sized to your capacity to review them, or rising continuously for 30 minutes: page.
• Zero consumers on the parking lot for ten minutes, or any consumer on a wait queue: alert.
• Source queue depth rising while the DLQ is flat, with at-least-once configured: the DLX or its target is broken.

Log at the moment of parking what the broker does not record: the exception class and message, the consumer host and build, the message id and correlation id, the original exchange and routing key from x-death, and the attempt count. The broker knows why a message was dead-lettered; only the consumer knows what went wrong. Attach the reason as a header too, as the example does with x-parked-reason, so whoever opens the queue later can read it without cross-referencing logs. Do not log full bodies unless they are small and contain nothing sensitive. The body is in the queue.

Replaying messages safely

Replay means publishing parked messages back to the work exchange. Fix the cause first. Replaying into an unfixed consumer creates a loop between the parking lot and the work queue with a human in the middle.

The simplest tool is a dynamic shovel from the parking lot to the work exchange, created with rabbitmqctl set_parameter shovel. Use ack-mode of on-confirm so a message leaves the parking lot only once the work queue has confirmed it, set dest-exchange to the work exchange and leave dest-exchange-key unset so the original routing key is used, and set src-delete-after to queue-length so the shovel drains what was present when it started and then removes itself. The management UI's move-messages action is a shovel under the hood. A shovel replays everything in order with no filtering, and x-death keeps accumulating, which is what you want: a message that fails again returns to the parking lot with a longer history.

When you need selection, use a replay consumer: a small tool that consumes from the parking lot with a modest prefetch, filters on header, time or id, publishes with confirms, and acks after each confirm. Rate-limit it, because a parking lot that filled over a weekend can stampede a downstream database on Monday morning. If you want attempt counts to start fresh it may strip x-death, but it should record an x-replay-count header in its place.

Do not republish from an ad-hoc script without confirms, do not replay during peak, and do not replay a batch whose root cause you have not identified.

Retention and disk

A parking lot is durable state. On a quorum queue every message is written to the Raft log and segment files on every member, so a three-member queue holds three copies for as long as the message sits there. Dead-lettered messages tend to be the large ones, and they tend to sit for days. Plan the disk.

Bound the parking lot with max-length-bytes and overflow set to reject-publish, not drop-head. drop-head silently discards the oldest evidence when the queue fills. reject-publish makes a full parking lot loud: dead-lettering into it fails, the at-least-once source queue holds its backlog, and your alerts fire. Loud is correct.

Do not set message-ttl on the parking lot. Retention belongs to an archival consumer that moves messages older than your review window to object storage or a database table and acks them, keeping the queue an inbox rather than an archive. The failure you are avoiding is real: a parking lot nobody drains grows until the node crosses the free disk alarm threshold, and the disk alarm blocks every publisher on the node, turning a slow accumulation of failures into a cluster-wide outage.

Anti-patterns we see in production

A DLQ with no consumer and no alert. Messages accumulate until the disk alarm fires or someone finds the queue during an unrelated incident months later. If you would not notice the queue growing, you have configured a leak with extra steps.

Dead-lettering into the exchange the queue is already bound to, with no change of routing key. The message lands back in the same queue. If it was expired or dropped for length, cycle detection discards it. If it was rejected, it hot-loops through the consumer at full speed, saturating CPU and filling logs.

Unbounded retries. With no attempt cap, a message that will never succeed circulates forever, costing consumer capacity on every lap, and with at-least-once and reject-publish it can back up into the source queue.

Mirrored classic queues carrying the DLX in 4.x. Classic queue mirroring was removed in RabbitMQ 4.0 and ha-* policy keys are no longer honoured. A dead letter queue that was replicated on 3.x becomes a single-node classic queue after the upgrade, and if that node fails the evidence goes with it. Convert dead letter queues to quorum queues before the upgrade.

TTL on the DLQ itself. A message-ttl of a week on the parking lot "to keep it tidy" deletes exactly the messages an investigation needs, and if the parking lot also dead-letters back to the work queue, the TTL builds a pure-expiry cycle that cycle detection drops silently.

Two smaller ones: logging the body and acking instead of parking, which treats a log line as a queue; and running the parking-lot consumer inside the same process as the work consumer, so the failure that fills the parking lot also takes down the thing meant to drain it.

Where to go next

Dead-lettering sits on top of routing, and a DLX is only as sensible as the exchange design around it. Our guide to exchanges, queues and bindings for microservices owns that topic. If you have not settled the queue-type question, classic versus quorum queues covers the decision; everything here assumes quorum queues for anything that matters. For the broader failure modes that fill a parking lot in the first place, see the top ten RabbitMQ troubleshooting issues.

If your dead letter queues are filling and you are not sure why, or you are planning the 4.x upgrade and want the DLX topology reviewed before mirrored queues stop being an option, that is what we do. AceMQ is Broadcom's exclusive strategic RabbitMQ MSP partner, supports RabbitMQ 3.8.x through 4.x, and answers P1 incidents within 15 minutes. See our RabbitMQ support plans or read more about how we work with RabbitMQ.