RabbitMQ Federation vs Shovel for DR: Which One, and When

Cross-region RabbitMQ is a message movement problem, not a replication problem

The question we hear most often from architects planning a second region is some version of "RabbitMQ federation versus shovel for DR?" It is the right question, but it hides a wrong assumption. Both plugins move messages between clusters that share no state. Neither copies a queue. Neither keeps two clusters in lockstep. If your DR plan expects the standby to hold the same messages as the primary at the moment it fails, federation and shovel will both disappoint you, and the disappointment will arrive at the worst possible time.

So, is RabbitMQ federation suitable for disaster recovery? Yes, for a specific shape of DR: one where you accept a bounded window of in-flight loss or duplication, design consumers to tolerate replay, and use the plugin to keep a second region warm with the traffic it needs. The same is true of shovel. This post explains what each plugin actually does in RabbitMQ 3.13 and 4.x, where their semantics differ in ways that matter for recovery, and how we combine them in a two-region design that survives a regional outage without inventing messages or losing them silently.

One scoping note. Cluster sizing, node counts and multi-AZ placement within a region are covered in our RabbitMQ HA and cluster sizing guide, and we will not repeat that material here.

How federation works: exchanges, queues, upstreams and hops

Federation is a pull model. A downstream cluster declares an upstream, opens an AMQP connection to it and asks for messages. The link is owned by the downstream, which is why the downstream is the side you configure, although we enable the plugin on both sides so roles can be reversed.

There are two flavours. Exchange federation makes an exchange on the downstream receive messages published to a same-named exchange on the upstream. Under the hood the downstream creates an internal queue on the upstream, binds it with the same bindings that exist on the downstream exchange, and consumes from it. Only messages matching a downstream binding cross the link, so a downstream with no bindings pulls nothing. Queue federation makes a downstream queue pull from a same-named upstream queue, but only when the downstream queue has consumers and the upstream queue has messages ready. It balances load between sites rather than copying: a message that moves downstream is gone from the upstream queue.

The max-hops parameter on the upstream definition controls how far a message propagates through a chain of federated exchanges. The default is 1: a message crosses one link and stops. Federation stamps an x-received-from header on each hop and will not forward a message back to an upstream it already visited, which is what prevents loops in a bidirectional setup. Leave max-hops at 1 unless you are deliberately building a mesh.

When the upstream is unreachable the downstream carries on serving local publishers and consumers. The link retries every reconnect-delay seconds, five by default, until the upstream returns. Messages published on the upstream during the outage accumulate in the internal upstream-side queue, bounded only by the message-ttl and expires you set on the upstream definition, and drain when the link returns. That is federation's most useful property for DR and its most common trap, because an unbounded upstream queue during a long outage can push the upstream into a disk alarm.

Ordering is preserved on a healthy link because the internal queue is FIFO with one consumer. It is not guaranteed across a reconnect, and the default ack-mode of on-confirm can redeliver messages that were in flight when the link dropped. Federation gives you at-least-once across the link, never exactly-once. And to state it plainly: federation is not replication and not a mirror. A federated exchange on the standby knows nothing about messages sitting unconsumed in queues on the primary.

# On the downstream (standby) cluster
rabbitmqctl set_parameter federation-upstream east \
  '{"uri":["amqps://fed:REDACTED@east-a.example.internal:5671","amqps://fed:REDACTED@east-b.example.internal:5671"],"max-hops":1,"ack-mode":"on-confirm","message-ttl":3600000,"expires":86400000,"reconnect-delay":5}'

rabbitmqctl set_policy --apply-to exchanges federate-events "^events\." \
  '{"federation-upstream-set":"all"}'

The URI is a list on purpose. If the first upstream node is down the link tries the next, which is exactly what a DR link should do.

How shovels work: static, dynamic and ack modes

A shovel is a consumer and a publisher glued together inside the broker. It consumes from a source queue, publishes to a destination exchange or queue, and acknowledges on the source according to its ack-mode. Messages leave the source. That is the defining difference from exchange federation: a shovel moves, federation copies according to bindings.

Static shovels live in advanced.config and need a node restart to change, so they suit fixed plumbing. Dynamic shovels are runtime parameters, created with rabbitmqctl set_parameter shovel or the management API, and can be added, changed and deleted live. For DR we use dynamic shovels almost exclusively, because failover and failback are precisely the moments you need to reconfigure without bouncing a node. A dynamic shovel runs on one node; if that node fails the shovel restarts on another.

The ack-mode decides what a shovel can lose. With on-confirm, the default, the shovel acknowledges the source message only after the destination confirms it, so a crash between publish and confirm can produce a duplicate but never a loss. With on-publish the shovel acknowledges once the message is written to the destination connection, before the confirm, so a destination failure at the wrong instant loses it. With no-ack the shovel uses automatic acknowledgement on the source and loses whatever was in its prefetch buffer when anything fails. For DR traffic there is one correct answer and it is on-confirm.

Reconnect behaviour mirrors federation: on loss of either endpoint the shovel waits reconnect-delay seconds and retries indefinitely, unless src-delete-after tells it to stop once a queue is drained. That option is useful for one-off migrations and for failback, where you want a shovel to empty a queue and remove itself.

Shovels compose into fan-in and fan-out topologies. Fan-in is several shovels, one per source cluster, publishing into one destination exchange; we use it to pull regional audit or billing events into a central cluster. Fan-out is several shovels from one cluster, each reading its own source queue bound to the same exchange and delivering to a different destination. A single shovel cannot fan out because it consumes each message once.

rabbitmqctl set_parameter shovel orders-dr-to-west \
  '{"src-protocol":"amqp091","src-uri":"amqps://shovel:REDACTED@east-lb.example.internal:5671","src-queue":"orders.work.dr","src-prefetch-count":500,"dest-protocol":"amqp091","dest-uri":"amqps://shovel:REDACTED@west-lb.example.internal:5671","dest-queue":"orders.work.standby","ack-mode":"on-confirm","reconnect-delay":5}'

What neither plugin gives you: synchronous replication

Inside a single cluster, quorum queues replicate every message to a majority of their members using Raft before confirming to the publisher. A confirmed message survives the loss of a minority of nodes with no gap and no replay. That is synchronous replication, and it is why quorum queues are the default for durable workloads within a region.

Neither federation nor shovel does anything like this. Both are asynchronous consumers running behind the primary's confirm path. The publisher gets its confirm when the primary's quorum queue commits; the standby receives the message some time later, after the link consumes it, ships it across the WAN and the standby confirms it. The gap is your exposure. On a healthy link it is usually tens of milliseconds, but it is never zero, and during a link outage it grows until the link recovers.

The commercial Tanzu RabbitMQ distribution from Broadcom includes a warm standby replication feature that replicates schema and message data from an active cluster to a passive one as a product capability with its own operational model. It exists, it is a legitimate answer for some organisations, and it is not part of the open source distribution. We will not restate its details here because they belong in the vendor documentation, but if a fully supported cluster-to-cluster replication feature is what your compliance posture demands, that is the conversation to have, and as Broadcom's strategic RabbitMQ MSP partner it is one we have regularly.

For everyone else the honest framing is this: RabbitMQ DR across regions is asynchronous, and the design work is about bounding the exposure and making the application indifferent to it.

Decision table: choosing between federation and shovel

The choice is rarely one or the other for a whole estate. It is per exchange and per queue, driven by what the messages mean and what consumers can tolerate.

Read the table as defaults. Events that many services subscribe to, where every region should see every event, belong on federated exchanges. Commands and jobs, where exactly one worker should act, belong in queues that are processed locally or shoveled deliberately to one destination.

How we design multi-region RabbitMQ: a reference two-region layout

Clients ask "How do we design multi-region RabbitMQ?" expecting one diagram. The honest answer is two overlays on the same pair of clusters, one for events and one for work.

Take two regions, East as primary and West as standby, each running its own three-node cluster with quorum queues for everything durable. Both clusters carry identical definitions: exchanges, queues, bindings, policies and users, exported from East and applied to West by the deployment pipeline so drift is impossible. If the basic exchange and binding layout is still being settled, our post on exchanges, queues and bindings for microservices covers it, and the production RabbitMQ architecture guide shows where this fits in the wider picture.

The event overlay uses exchange federation in both directions. Every topic exchange named events.* is federated, West as downstream of East and East as downstream of West, with max-hops set to 1. Services that subscribe to events run in both regions and consume from local queues bound to the local federated exchange. In normal operation East publishes nearly everything, West receives copies and its subscribers process them (read models, caches, search indexes), and the reverse link sits idle. Because bindings drive what crosses the link, a subscriber that exists only in East does not drag its traffic into West.

The work overlay is where shovels earn their place. Each command queue in East, say orders.work, is fed by a binding from an orders.cmd exchange. A second binding routes the same messages to orders.work.dr, a quorum queue with a message-ttl matched to the longest an order should legitimately sit unprocessed in East, typically fifteen to thirty minutes. A dynamic shovel with on-confirm continuously moves orders.work.dr into orders.work.standby in West, which carries the same TTL. Nothing consumes orders.work.standby in normal operation; messages arrive, age out and vanish. The queue is a rolling window of recent work, not a mirror of the primary's backlog.

Command consumers run only in East during normal operation. West has the deployment ready but scaled to zero for command consumers, and running for event subscribers. Publishers connect through a DNS name or global load balancer entry that resolves to East.

# Applied on both clusters via the definitions pipeline
rabbitmqctl set_policy --apply-to queues dr-window "^orders\.work\.(dr|standby)$" \
  '{"message-ttl":1800000}'

Surviving a regional outage: failover and failback without duplicates or loops

"How should messages survive regional outages?" breaks into three moments: the outage, the failover decision, and the failback.

During the outage the plugins need no attention. West's federation link to East fails and retries every five seconds. The shovel from East cannot connect and retries. West's event subscribers keep working on anything already delivered. Messages published in East in the seconds before the failure that had not yet crossed the link are stuck there until East returns; that is the RPO, and the application layer must own it through publisher-side outboxes or replay from the system of record.

Failover is a deliberate action, not an automatic one. We have watched automated cross-region failover triggered by a monitoring blip create a split-brain of consumers that took a weekend to reconcile. The runbook is short: confirm East is genuinely unavailable through independent probes; repoint the publisher DNS or load balancer entry to West; scale West's command consumers up from zero; confirm they are draining orders.work.standby. Everything in that queue younger than the TTL gets processed, which means work East had already completed in the last window is processed again. This is by design. Every command consumer carries an idempotency key, and the TTL bounds how much duplicate work the standby can ever see.

West is now primary. Its events.* traffic is what matters, and the East-as-downstream federation link will drain it to East when East returns. Commands in West land in orders.work and are processed locally, with a copy routed into West's orders.work.dr, ready for the reverse shovel.

Failback is where duplicates and loops get created if you rush. The sequence we use: bring East up with no publishers pointed at it and its command consumers at zero; let the federation link from West to East drain the event backlog; create a reverse shovel, on-confirm, from West's orders.work.dr into East's orders.work.standby; then in a quiet period repoint publishers to East, scale East's consumers up, scale West's down, and remove the reverse shovel with src-delete-after set to queue-length so it exits once empty. At no point are two shovels moving the same queue in opposite directions, and at no point do command consumers run in both regions against the same logical queue.

Loops cannot form on the event overlay because max-hops of 1 and the x-received-from header stop a message that has crossed one link from crossing back. Loops can form on the work overlay only if someone shovels into an exchange whose bindings feed the shovel's own source queue. The rule: shovel into named destination queues, never into the exchange that fed the source.

Failure drills to run before go-live

A DR design that has never been exercised is a hypothesis. We run these drills with clients before the first production cutover and again each quarter.

Block the WAN port between the clusters at the firewall for thirty minutes and watch the upstream-side federation queue and orders.work.dr grow. Confirm the message-ttl and expires you set actually bound the growth and that neither cluster approaches its disk alarm threshold. Restore the link and measure how long the backlog takes to drain at your real prefetch settings.

Publish a known count of numbered messages while the link is down, restore it, and count what arrived. This is the drill that finds no-ack shovels and on-publish links that someone set "for performance".

Restart the node that owns the federation link and the node that owns the shovel, separately, and confirm both re-establish elsewhere. Check rabbitmqctl federation_status and rabbitmqctl shovel_status after each restart rather than glancing at the management UI.

Trigger a memory alarm on the standby by deliberately filling a queue. A federation link publishes into the downstream, so a downstream alarm blocks the link exactly as it blocks any other publisher, and the upstream queue starts growing. Confirm your alerting catches the blocked link, not only the alarm.

Run the full failover runbook against the standby with real consumers and synthetic traffic, then the full failback. Count duplicates processed in each phase and confirm the idempotency layer discarded all of them. Time both procedures and write the numbers into the runbook, because those are your real RTO figures.

Rotate the credentials in the upstream and shovel URIs and confirm the links reconnect. Expired federation credentials discovered during a real outage is a failure mode we have seen more than once.

Mistakes we see in production

The most common mistake is treating federation as replication and discovering during an incident that the standby holds none of the primary's unconsumed backlog. The fix is architectural: work that must survive needs the DR buffer queue and shovel pattern, or an application-level outbox.

Close behind is an upstream definition with no message-ttl and no expires. The internal federation queue on the upstream is durable and grows through a long link outage until the upstream hits a disk alarm and stops accepting publishes from everyone, turning a standby problem into a primary outage.

Shovels set to no-ack or on-publish because a benchmark showed higher throughput. The throughput is real and so is the loss. If a shovel needs to go faster, raise src-prefetch-count and keep on-confirm.

Bidirectional shovels on the same queue, usually created during a hurried failback and never removed. They pass the same messages back and forth until someone notices the counters.

A single hostname in the upstream URI. When that node is the one that failed, the standby's link has nowhere to go even though the rest of the primary cluster is healthy. Supply a list of URIs or a load balancer address that health-checks the nodes.

Definitions drift between regions. A queue that exists in East but not in West leaves the shovel in a retry loop, or worse, publishing into an exchange with no bindings where messages are dropped. Export definitions from the primary and apply them to the standby in the pipeline that deploys the application.

No monitoring of link state. Nobody watches the management UI at three in the morning. Export federation_status and shovel_status into your metrics pipeline and alert on any link that is not running for longer than the reconnect delay.

Finally, DNS TTLs and connection pools that keep publishers pinned to the failed region long after the record changed. Client libraries reconnect to the address they resolved at startup unless told otherwise, so the failover runbook must include a client restart, or clients must be configured with both regions' endpoints.

Where to go next

This post has deliberately stayed on the wire between clusters. How large each cluster should be, how to spread nodes across availability zones and what quorum queue settings to use are owned by our RabbitMQ HA and DR cluster sizing guide. The messaging topology the federation and shovel overlays sit on is described in exchanges, queues and bindings for microservices, and the end-to-end shape of an enterprise deployment is in the production RabbitMQ architecture guide.

If you run RabbitMQ 3.8.x through 4.x in production and want a second region that has actually been failed over and failed back rather than drawn on a whiteboard, AceMQ RabbitMQ support gives you a 15-minute P1 response from engineers who have done it, and our RabbitMQ consulting engagements design, build and drill the setup with your team before it has to work for real.