What is the default retention policy in Kafka?

Apache Kafka's default cleanup policy is delete with a retention period of 168 hours, or seven days, controlled by log.retention.hours at the broker level. Size-based retention limits are switched off out of the box: for log.retention.bytes the default is -1. Any topic created without explicit overrides inherits these values. Most production deployments override them per topic, because a single cluster-wide setting rarely fits every workload.

Why is my Kafka data still there after the retention period?

Retention applies only to closed log segments, never to the segment currently being written. If a topic has low throughput, that segment may not reach segment.bytes (1 GB by default) or segment.ms for a long time, so no data becomes eligible for cleanup. Reducing segment.ms or the segment size forces segments to roll more often and lets cleanup take effect. Compacted topics behave differently again, since a backlogged log cleaner leaves old records in place.

What is the difference between delete.retention.ms and retention.ms?

retention.ms sets how long records are kept on a topic that uses the delete cleanup policy. delete.retention.ms applies only to compacted topics, where it governs how long tombstone markers survive after compaction so consumers can observe that a key was removed. They are separate mechanisms and do not substitute for one another. Setting it too low causes consumers to miss removals and drift out of sync with the source topic.

Can I set different retention for different topics?

Yes. Topic-level configuration overrides the cluster-wide value, so each topic can carry its own retention.ms, retention.bytes and cleanup.policy. Set them at topic creation or modify them later with kafka-configs.sh --alter --entity-type topics. This is the recommended approach, because compliance data, changelogs and debug streams have very different requirements.

Does increasing Kafka retention hurt performance?

Longer retention does not slow consumers reading recent data, which is served from the operating system page cache. It does increase storage usage, the number of open file handles, and the time a server needs for log recovery after an unclean restart. Partition reassignments also take longer because more data has to move. The practical cost is storage and recovery time rather than steady-state throughput.

Kafka Retention Policy: Time-Based & Size-Based Config

A Kafka retention policy defines how long a topic keeps messages before they become eligible for deletion, and it is set through two independent controls: cleanup.policy, which chooses between discarding old records and compacting them to the latest value per key, and the limits retention.ms and retention.bytes, which set the time and size thresholds. The default is 168 hours (seven days), applied per partition. Messages in Kafka are removed a whole log segment at a time, never message by message, which is why data often outlives the window you configured.

How Long Can a Kafka Topic Store Data?

Indefinitely, if you configure it that way. Setting retention.ms=-1 disables time-based retention and retention.bytes=-1 disables the size-based equivalent, so the topic retains messages until you change it or exhaust the disk. The ceiling is storage, not software: a full volume takes the whole cluster down. Tiered storage offloads closed segments to object storage, making longer retention periods affordable without oversizing local disks.

Log retention controls how long messages remain readable, not how long consumers have to read them. A consumer group that falls behind the window loses those records for good.

delete.retention.ms Versus retention.ms

These govern two entirely different retention mechanisms, and conflating them is a common and costly mistake. retention.ms is the time-based limit on a topic using the delete policy: records older than that become eligible for deletion. delete.retention.ms applies only to compacted topics, where it controls how long tombstones — null-valued records marking a key as removed — survive after compaction. It defaults to 24 hours. Consumers need that window to observe the removal and propagate it; set it below your slowest consumer's catch-up time and you get silent data loss downstream. On a purely compacted topic, retention.ms does not age records out at all.

The Three Cleanup Policy Options

There are three values for cleanup.policy:

delete (the default) discards entire segments once they exceed the time or size threshold — right for event streams, telemetry and logs.
compact retains the most recent value for every key and removes superseded versions. Log compaction turns the topic into a durable changelog for rebuilding state.
delete,compact applies both: compaction keeps the latest value per key while a retention period still ages out old segments. The internal __consumer_offsets topic uses this combination.

Configuring Retention at the Broker and Topic Levels

Retention configuration exists at two levels. Broker configuration sets cluster-wide values through log.retention.hours (168), log.retention.bytes, log.segment.bytes and log.cleanup.policy, and that log retention is what every topic inherits by default. Topic-level configuration uses the equivalents retention.ms, retention.bytes, segment.bytes and cleanup.policy, set at topic creation or modified later on a running cluster. Topic level always wins: topic retention overrides the server setting, giving granular control over retention per workload. When you create a topic without overrides, it simply takes the broker defaults.

retention.bytes is enforced per partition, not per topic — a 100 GB limit on a 12-partition topic permits 1.2 TB of disk space, tripled at a replication factor of three.

Adjusting Retention Settings on a Running Cluster

Topic configuration is dynamic, so you can configure retention on a live topic without restarting anything:

kafka-configs.sh --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=604800000

The new value applies to segments already on disk, not only to future ones. Raise a window freely; adjust retention settings downwards only when you are certain the data is expendable, because the next cleanup pass will act on it and that can lead to data loss you cannot undo.

Log Cleanup and Segment Management

Each partition is an append-only commit log split into log segments, and only a closed segment is a cleanup candidate. On disk each one is a .log file sitting beside its index files. Writes land in the active segment, which rolls when it reaches segment.bytes (1 GB by default) or segment.ms (seven days), whichever comes first. Kafka brokers then evaluate every closed log segment file against the retention settings and unlink those that qualify, so the time limit is applied per segment rather than per record.

That mechanism explains why data survives past its retention window. A low-traffic topic can take weeks to fill a 1 GB segment, so nothing rolls, nothing becomes eligible for cleanup, and a seven-day window behaves like a seven-week one. Lowering segment.ms or the size of the log segments restores the behaviour you expected.

Compacted topics are handled by dedicated log cleaner threads (log.cleaner.threads, default 1). min.compaction.lag.ms guarantees a minimum age before a record can be compacted away, so consumers of real-time data still see every update. An under-provisioned cleaner falls behind and leaves duplicate keys in place.

How Retention Affects Storage and Performance

Retention is the dominant driver of storage costs: ingest rate times retention time times replication factor gives the floor, so 100 MB/s for seven days at RF=3 is roughly 180 TB. Managing storage costs therefore starts with the window rather than with the hardware order. Longer windows do not slow reads of recent data, which is served from page cache, but they multiply open file handles and lengthen log recovery. Cleanup itself is a file unlink; compaction consumes real CPU and I/O.

Best Practices and Common Challenges

Set retention per topic rather than one cluster-wide value; audit and financial topics rarely need the same window as debug telemetry, and per-topic windows are the only practical way to honour differing data retention policies.
Pair the time and size-based retention limits: time protects compliance obligations, size protects the volume, and whichever triggers first wins. Expiring segments based on time alone leaves you exposed to a traffic spike.
Align segment.ms with your shortest window so segments actually roll.
Enable log compaction only where the topic really is a keyed changelog; on a plain event stream it is wasted CPU.
Never treat a compacted topic as a backup: tombstones expire, and delete.retention.ms decides when.
Change retention settings with kafka-configs.sh, not by editing server files, then watch log-cleaner metrics.

Shortening a window on a live topic is irreversible — one of the few mistakes that destroys data rather than merely delaying it. AceMQ's 11+ senior Kafka SMEs review these settings alongside capacity planning for 130+ enterprise customers across 26+ countries. If a cluster is filling up right now, 24/7 Kafka support carries a 15-minute emergency response SLA, alongside the full range of enterprise Kafka services.