The RabbitMQ Streams Guide

A stream is a persistent, replicated, append-only log that sits in RabbitMQ next to classic and quorum queues. The difference is in how messages leave it: they do not. Consuming from a queue removes the message; consuming from a stream only moves that consumer's position, so any number of consumers can read the same messages, as many times as they want, until retention expires them. The RabbitMQ documentation calls this non-destructive consumer semantics. Streams are always durable and always replicated; there is no transient variant.

Streams were built for four cases the queue types handle badly: large fan-out (many consumers reading the same messages without a queue each), replay (attach at any point in the log and read from there), throughput (the design goal was to compete with log-based systems), and large backlogs (queues are optimized to converge on empty; streams store millions of messages with minimal memory).

They are the wrong type when you need queue features. Streams do not support dead letter exchanges, message TTL, queue length limits, message or consumer priority, exclusivity or non-durable declaration, and the documentation is explicit that many of these will never be supported because of the non-destructive read model. If the workload is work distribution with retries and dead-lettering, that is a quorum queue.

Declare a queue with the x-queue-type argument set to stream. The type must be given by the client at declaration; it cannot be set or changed by policy. A stream is still an AMQP 0.9.1 queue, so it binds to any exchange like one.

Set retention on day one, because an append-only log grows until the disk is full. Two settings, usable together and changeable by policy: max-age (units Y, M, D, h, m, s, so 7D is a week) and max-length-bytes. Retention is evaluated per segment file, and the default segment size is 500 MB, so a stream always keeps at least one segment and will sit above its configured limit by up to a segment. Size the disk for the limit plus a segment per stream, not the limit alone.

A consumer chooses where to start with the x-stream-offset argument: first, last (the last written chunk), next (the default: only new messages), a numeric offset, a timestamp, or an interval such as 1h. Timestamps resolve to chunk boundaries, so a consumer can receive messages published slightly before the time it asked for.

Two rules catch teams moving from queues. Consumers must set a QoS prefetch, and acknowledgements are required: on a stream the ack acts as a credit that advances the consumer's offset. And global QoS is not supported; consuming from a stream on a channel with global prefetch returns a channel error. Use per-consumer prefetch.

Any AMQP 0.9.1 client that can pass queue and consumer arguments can use a stream. The RabbitMQ team still recommends the dedicated binary stream protocol and its clients, for two reasons: it gives the best throughput, and some features exist only there. Broker-side offset tracking is one: it is available only through the stream plugin, and offsets are stored in the stream itself as non-message data. Publisher deduplication is another: a named producer with a strictly increasing publishing ID lets the broker filter re-sent messages after a crash, provided only one producer instance uses that name at a time.

One side effect to know: messages are stored as AMQP 1.0 data, so AMQP 0.9.1 headers with complex values such as arrays or tables are not converted.

A super stream is a logical stream partitioned across several regular streams, which spreads storage and traffic over cluster nodes. Client libraries present it as one stream. Combined with single active consumer, where only one instance in a named group receives messages and another takes over if it stops, you get ordered processing per partition with failover. Both arrived in RabbitMQ 3.11. The CLI creates the topology in one line: rabbitmq-streams add_super_stream invoices --partitions 3.

The documentation's own advice is restraint: super streams add complexity and should be used only once a single stream has reached its limits.

Streams are quorum systems, so use odd cluster sizes: three nodes tolerate one failure, five tolerate two, and a two-node cluster tolerates none. Replica membership is managed by the operator. A node added to the cluster hosts no stream replicas until you run rabbitmq-streams add_replica, and a node being decommissioned must be removed with delete_replica from every stream it hosts. Check replication with rabbitmq-streams stream_status. When replacing a node, add the new one, wait for it to be in sync, then remove the old one; a replica cannot be added while another is out of sync.

Streams use little RAM in the broker but lean heavily on the kernel page cache. In containers that page cache counts toward the memory limit, which is how a healthy stream workload gets a pod OOM-killed. Streams also do not react to memory alarms. Use the fastest disks available; throughput falls as message size and replica count rise.

A confirm on a stream means the message has been replicated to a quorum of replicas. It does not mean it has been flushed to disk: streams write to disk but do not fsync, leaving that to the operating system. An uncontrolled shutdown can therefore lose recent data on that node, which is normally re-replicated from the others. The documentation's guidance is direct: if you need stronger guarantees, use quorum queues, which confirm only after a quorum has written and flushed. Messages published without confirms carry no guarantee at all.

Check the platform before the design. Amazon MQ for RabbitMQ does not support streams, and its documentation warns that creating one will result in data loss. Streams need self-managed RabbitMQ, a specialist host, or brokers in your own account operated for you.