Redis vs Kafka Compared

Redis offers three ways to move messages. Pub/Sub has at-most-once delivery: a subscriber that is offline or fails mid-message never sees it again. Lists work as simple job queues. Streams are an append-only log with random access by ID and consumer groups, which makes them the only Redis structure comparable to Kafka.

A Kafka topic can have many producers and many subscribers, and events are not deleted when they are read. Each topic is split into partitions spread over brokers, and events with the same key always go to the same partition. Kafka architecture covers brokers, partitions and replication in depth.

Redis persists an in-memory dataset in two ways. RDB snapshots are usually taken every five minutes or more, so the Redis persistence guide says to expect to lose the latest minutes of data after an unclean stop. The append-only file with appendfsync everysec narrows that to about one second. RDB's fork() can pause Redis for milliseconds, or up to a second on a very large dataset. Replication is asynchronous, and the Redis Cluster documentation states that it can lose writes it has acknowledged to the client.

Kafka writes every message to a log on disk, though not necessarily flushed, and relies on replication for safety. A message is committed only when all in-sync replicas for its partition have applied it, and a committed message is not lost while one broker holding a replica remains alive. The Kafka documentation calls a replication factor of three a common production setting.

A Redis stream keeps entries until it is trimmed, either with MAXLEN for a fixed length or MINID for entries older than an ID. Replay with XRANGE works for whatever is still in memory. Kafka retention is a per-topic setting: log.retention.hours defaults to 168 and log.retention.bytes is unlimited by default. Kafka's documentation says its performance is effectively constant with data size, so long retention is normal. Log compaction keeps the latest value for each key instead of deleting by age.

Redis borrowed the term from Kafka, and its documentation says plainly that Redis consumer groups have nothing to do with Kafka's in implementation. With XREADGROUP, each consumer in a group receives different entries, each delivered entry stays in the group's Pending Entries List until XACK, and XAUTOCLAIM hands a dead consumer's entries to a live one. Redis 8.6 added idempotent production, so producer retries do not add duplicate entries.

In Kafka, each partition is read by exactly one consumer in each group, and progress is a single offset per partition. That makes acknowledgement cheap and keeps order per partition, but parallelism inside one group stops at the partition count. Kafka 4.3 also documents share groups, where several consumers can read one partition and acknowledge records individually, which is closer to a work queue.

A Redis stream is totally ordered by entry ID, but it is one key, and in Redis Cluster every key maps to one hash slot on one primary. One stream therefore scales with one node; more throughput means sharding across several stream keys in application code. Kafka orders events within a partition and scales a topic by adding partitions and brokers. Its design sends data from the page cache straight to the network, which the documentation says lets consumption approach the limit of the network connection.

Benchmarks rarely compare like with like. Test Redis with the persistence your loss tolerance needs against Kafka with the replication and producer acknowledgements you would run, on your own message sizes.

Every Redis stream entry occupies RAM until trimmed, so a consumer outage grows memory use. Size Redis for the peak backlog, not the average, and keep queue data away from eviction policies meant for cache keys. Kafka keeps the backlog on disk, which costs less per gigabyte, so a slow consumer costs disk space rather than an out-of-memory incident. The trade is operating a broker cluster.

Kafka Connect is Kafka's tool for streaming data between Kafka and other systems, from whole databases in to downstream stores out. Kafka Streams is a client library for building applications whose input and output live in Kafka, and transactions give read-process-write pipelines exactly-once semantics. Redis has no built-in counterpart for streams, so pipelines around Redis Streams are application code.

The two often work together. Kafka holds the event history and feeds downstream systems; a consumer keeps Redis up to date as a cache, session store, leaderboard or rate limiter that answers from memory. A lost Redis can be rebuilt by replaying the topic. What Redis is used for lists the jobs Redis does best, and RabbitMQ vs Redis covers the case where the real need is a work queue rather than an event log.

1. Do several teams or systems need to read the same events, now and later? Kafka.
2. Must events survive for days, or be replayed into a new system? Kafka.
3. Is Redis already in production, with a short window of events that fits in memory? Redis Streams with consumer groups.
4. Is the message a notification nobody needs to recover? Redis Pub/Sub.
5. Do you need fast reads of current state built from events? Both: Kafka for the log, Redis for the view.