RabbitMQ Streams vs Kafka: Which One Your Architecture Actually Needs

Both give you a replayable, append-only log that many consumers can read independently. The difference is what surrounds it: RabbitMQ brings sophisticated routing and a broker your team probably already runs, while Apache Kafka brings raw scale, long retention, and a large streaming ecosystem. For most teams the deciding factor is not benchmark numbers — it is whether you need message routing or data volume, and which system you can operate well.

The short version: if you already run RabbitMQ and need replay, streams are very likely the answer. If you are ingesting hundreds of thousands of messages per second and retaining them for weeks, Kafka is.

A classic RabbitMQ queue is destructive: a message is delivered, acknowledged, and gone. That model suits work distribution — one consumer takes a job, does it, and it disappears.

A stream is the opposite. It is an append-only log on disk. Consumers read at their own offset and reading changes nothing, so ten independent applications can each read every message and replay from the beginning whenever they need to.

That non-destructive model is what the two systems share, and it is why they get compared at all. Everything else about them differs.

Kafka is a distributed event streaming platform, and that phrase is doing real work. A Kafka topic is split into partitions spread across the cluster; a producer writes to a partition, and consumers in a group each own some partitions. Parallelism, ordering, and throughput all derive from that partition model.

RabbitMQ is a message broker that gained a stream queue type. The same cluster that routes work through exchanges and bindings can also hold streams, reachable over AMQP or a dedicated binary stream protocol. RabbitMQ supports both models in one place — you are adding a capability to a broker, not adopting a second platform.

That single structural fact explains most of the practical differences below.

Both retain data on disk and let consumers replay it. A Kafka topic retains messages with time-based, size-based, or compacted policies, and compaction — keeping only the latest value per key — is genuinely distinctive; RabbitMQ streams do not offer it.

Ordering works the same way in both: guaranteed within a partition, not across them. If strict global ordering matters, you need a single partition in either system, and you give up parallelism to get it.

The difference appears in routing. RabbitMQ can apply exchange bindings, topic patterns, and headers before a message ever reaches a stream. A Kafka producer chooses a topic and a partition key, and routing logic lives in your application or in stream processing downstream.

Kafka throughput wins outright, and the gap is real at the top end. A Kafka broker is built for sequential disk writes, and the partition model scales writes horizontally in a way a single stream does not.

RabbitMQ streams are far faster than classic queues and handle high-volume workloads comfortably — but a super stream is partitioning bolted onto a broker whose center of gravity is routing, not a platform engineered from the ground up for sequential disk throughput.

Latency is closer than throughput. For per-message latency at moderate volume, RabbitMQ is frequently the better performer, because Kafka's batching optimizes for bulk over individual message speed. If your requirement is "this message must arrive in single-digit milliseconds" rather than "we must absorb a million events a second", do not assume Kafka is the faster choice.

Benchmarks in either direction should be treated carefully. Vendor numbers are produced under conditions chosen to flatter the system being sold, and your message size, durability settings, and network will move the result more than the software choice.

RabbitMQ, generally — and that is a legitimate deciding factor rather than an admission of weakness.

The RabbitMQ deployment model maps closely to how most developers already think: producers, exchanges, queues, consumers. RabbitMQ brokers behave predictably. Client libraries exist for every language, the management UI is genuinely usable, and a working cluster is achievable in an afternoon.

Kafka asks more. Partition counts, consumer group rebalancing, offset management, retention tuning, and replication settings all have to be understood before production. The payoff is a system that scales further than RabbitMQ will — but the learning curve is real, and

The honest framing: choose the system your team can operate at 3am, not the one that wins a benchmark.

Partially, in both directions, and badly in both directions if forced.

Kafka replacing RabbitMQ struggles wherever you need routing. Kafka has no equivalent of exchange bindings, per-message TTL, priority queues, or dead-letter behavior. Reproducing those means building them in application code — a lot of work to avoid running a broker you already understand.

RabbitMQ replacing Kafka struggles at sustained volume and retention. If you are keeping a month of events for reprocessing at hundreds of thousands of messages per second, streams are not the right tool, and the ecosystem gap matters too: Kafka Streams, Connect, and ksqlDB have no direct counterpart.

Both can do the other's job for a while. The failure arrives at scale or at complexity, which is exactly when a migration is hardest.

Plenty of mature architectures run both, and it is not a failure of design.

A common shape: RabbitMQ handles command and control — request/reply, task distribution, event-driven microservices needing precise routing — while Kafka carries the high-volume event firehose that analytics and stream processing consume. Each system does what it is good at, connected by a bridge or an application that reads from one and writes to the other.

The cost is two systems to operate, monitor, and staff. Worth it when the workloads genuinely differ; not worth it to avoid one difficult conversation about which to standardize on.

Use RabbitMQ when:

• You already run RabbitMQ and need replay or multiple independent readers
• Routing matters — topic patterns, header-based routing, complex topologies
• You need queues and streams and would rather not run two platforms
• Per-message latency matters more than aggregate volume
• Your team is small and operational simplicity has real value

Use Kafka when:

• Ingest volume is genuinely very high and sustained
• You need long retention, replay across weeks, or log compaction
• Your streaming use cases need the wider ecosystem — Connect, Streams API, ksqlDB
• Many independent teams consume the same data streams
• You have, or will hire, the operational expertise it requires

Choose both when you have distinct workloads that genuinely suit each, and the operational capacity to run them properly.

Worth stating plainly: if your workload is task distribution rather than event replay, a quorum queue remains the correct choice. Streams add retention and replay you may not need, and they store everything on disk, so a stream holding data nobody replays is expense without benefit.

The queue-versus-stream decision inside RabbitMQ comes first, and often removes the Kafka question entirely. See

RabbitMQ streams performance: what to expect

A RabbitMQ stream is an append-only log on disk with one writer, the leader member. That single writer is why a stream is fast: no per-message routing, no queue index churn, no deletion on ack. It is also the ceiling. Throughput scales per stream up to what one leader can append and replicate, and beyond that with the number of streams spread across the cluster. Need more? Add streams, not a bigger box.

How you talk to the stream matters more than almost anything else. The native stream protocol on port 5552 (the rabbitmq_stream plugin) is a binary protocol built for logs: publishers send batches in one frame, the broker writes chunks straight to the file, and consumers read those chunks back with sendfile from the page cache with almost no per-message work in Erlang. AMQP 0-9-1 access to the same stream (x-queue-type: stream) goes through the regular channel machinery: one delivery per message, mandatory prefetch, per-message acks. It is convenient for existing code and an order of magnitude or more slower on reads. With the stream protocol and batching, a single stream on ordinary hardware reaches hundreds of thousands of messages per second, and a cluster with several streams reaches into the millions. Over AMQP, tens of thousands per second per stream is the realistic range.

Sub-entry batching pushes the native protocol further: a publisher packs many messages into one entry and compresses it (gzip, snappy, lz4, zstd), and the broker stores the batch as a unit, so disk and replication cost is paid once per batch. AMQP consumers cannot read sub-entry batches.

Replication is leader plus replicas using the Osiris log replication library rather than Raft. A publish is confirmed once a majority of members have written it to disk. Extra replicas cost the leader outbound bandwidth and add the slowest replica's disk latency to confirm time; they do not cost read throughput, since consumers can read from replicas.

Disk behaviour is the part people underestimate. Writes are sequential and cheap. Reads of recent data come from the OS page cache and are close to free; a consumer replaying from last week goes to disk and competes with the writer. Long retention plus rewinding consumers needs page cache headroom: RAM for the kernel, not the Erlang VM.

Retention is enforced per segment. max-age and max-length-bytes set the window, stream-max-segment-size-bytes (default 500 MB) sets the granularity, because the broker only deletes whole segments. Neither affects publish speed much; they decide disk usage and replay cost.

Offset tracking is not free either. Server-side offset storage writes tracking entries into the stream itself, so committing after every message roughly doubles a consumer's write load. Commit on an interval and accept a few redeliveries on restart.

How to tune RabbitMQ stream performance

In the order we reach for them:

1. Use the stream protocol and a stream client. If your hot path is on AMQP, this change is worth more than everything below combined.
2. Batch on the publisher, confirm asynchronously. Stream clients batch by default; raise the batch size and never wait for a confirm per message.
3. Give consumers credit. The protocol is credit based: the consumer asks for chunks and the broker sends until credit runs out. Keep several chunks outstanding.
4. Split with super streams. A super stream is a set of partition streams behind one name; publishers pick a partition by routing key, hashed by default. Ordering holds inside a partition and nowhere else, so choose a key that groups what must stay ordered (account, device, session) and spreads the rest. Partitions get different leaders, which is how you use every node.
5. Set replication deliberately. x-initial-cluster-size at declaration controls member count. Three for anything that matters, one for replayable scratch ingest.
6. Put members on fast local disks. NVMe over network-attached volumes, and not shared with quorum queue data.
7. Size retention for your replay pattern. Short retention if consumers stay near the head; modest retention and generous RAM if they rewind hours. Cluster sizing itself is covered in our HA and disaster recovery sizing FAQ.
8. Commit offsets on an interval. Every few seconds or every few thousand messages.
9. Keep AMQP consumers off high-throughput streams. Fine for a dashboard, wrong for the main consumer.
10. Watch the right metrics. The management UI shows per-stream publishers, consumers and the leader node; the Prometheus endpoint exports stream connection, publisher and consumer counts alongside queue-level message and byte rates. Consumer lag is a client-side number, so export it from the application.

A retention policy looks like this:

rabbitmqctl set_policy stream-events "^events\." \
  '{"max-age":"7D","max-length-bytes":50000000000}' --apply-to queues

The two mistakes we see most. First, one stream for everything: a single leader capped on one node, every consumer filtering client-side, and a replay that reads terabytes to find one tenant. Partition with a super stream or use several streams. Second, treating a stream like a queue: expecting per-message acks, competing consumers that share work, messages gone once handled. A stream keeps everything until retention, every consumer reads everything, and acks do not exist. If the workload wants queue semantics, it wants a quorum or classic queue, not a stream.

If your streams are slower than they should be, RabbitMQ support will read the configuration and metrics with you; RabbitMQ consulting covers redesign when the topology is the problem.