Kafka Retention Policy: Time-Based & Size-Based Config

A Kafka retention policy defines how long a topic keeps messages before they become eligible for deletion, and it is set through two independent controls: cleanup.policy, which chooses between discarding old records and compacting them to the latest value per key, and the limits retention.ms and retention.bytes, which set the time and size thresholds. The default is 168 hours (seven days), applied per partition. Messages in Kafka are removed a whole log segment at a time, never message by message, which is why data often outlives the window you configured.

How Long Can a Kafka Topic Store Data?

Indefinitely, if you configure it that way. Setting retention.ms=-1 disables time-based retention and retention.bytes=-1 disables the size-based equivalent, so the topic retains messages until you change it or exhaust the disk. The ceiling is storage, not software: a full volume takes the whole cluster down. Tiered storage offloads closed segments to object storage, making longer retention periods affordable without oversizing local disks.

Log retention controls how long messages remain readable, not how long consumers have to read them. A consumer group that falls behind the window loses those records for good.

delete.retention.ms Versus retention.ms

These govern two entirely different retention mechanisms, and conflating them is a common and costly mistake. retention.ms is the time-based limit on a topic using the delete policy: records older than that become eligible for deletion. delete.retention.ms applies only to compacted topics, where it controls how long tombstones — null-valued records marking a key as removed — survive after compaction. It defaults to 24 hours. Consumers need that window to observe the removal and propagate it; set it below your slowest consumer's catch-up time and you get silent data loss downstream. On a purely compacted topic, retention.ms does not age records out at all.

The Three Cleanup Policy Options

There are three values for cleanup.policy:

• delete (the default) discards entire segments once they exceed the time or size threshold — right for event streams, telemetry and logs.
• compact retains the most recent value for every key and removes superseded versions. Log compaction turns the topic into a durable changelog for rebuilding state.
• delete,compact applies both: compaction keeps the latest value per key while a retention period still ages out old segments. The internal __consumer_offsets topic uses this combination.

Configuring Retention at the Broker and Topic Levels

Retention configuration exists at two levels. Broker configuration sets cluster-wide values through log.retention.hours (168), log.retention.bytes, log.segment.bytes and log.cleanup.policy, and that log retention is what every topic inherits by default. Topic-level configuration uses the equivalents retention.ms, retention.bytes, segment.bytes and cleanup.policy, set at topic creation or modified later on a running cluster. Topic level always wins: topic retention overrides the server setting, giving granular control over retention per workload. When you create a topic without overrides, it simply takes the broker defaults.

retention.bytes is enforced per partition, not per topic — a 100 GB limit on a 12-partition topic permits 1.2 TB of disk space, tripled at a replication factor of three.

Adjusting Retention Settings on a Running Cluster

Topic configuration is dynamic, so you can configure retention on a live topic without restarting anything:

kafka-configs.sh --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=604800000

The new value applies to segments already on disk, not only to future ones. Raise a window freely; adjust retention settings downwards only when you are certain the data is expendable, because the next cleanup pass will act on it and that can lead to data loss you cannot undo.

Log Cleanup and Segment Management

Each partition is an append-only commit log split into log segments, and only a closed segment is a cleanup candidate. On disk each one is a .log file sitting beside its index files. Writes land in the active segment, which rolls when it reaches segment.bytes (1 GB by default) or segment.ms (seven days), whichever comes first. Kafka brokers then evaluate every closed log segment file against the retention settings and unlink those that qualify, so the time limit is applied per segment rather than per record.

That mechanism explains why data survives past its retention window. A low-traffic topic can take weeks to fill a 1 GB segment, so nothing rolls, nothing becomes eligible for cleanup, and a seven-day window behaves like a seven-week one. Lowering segment.ms or the size of the log segments restores the behavior you expected.

Compacted topics are handled by dedicated log cleaner threads (log.cleaner.threads, default 1). min.compaction.lag.ms guarantees a minimum age before a record can be compacted away, so consumers of real-time data still see every update. An under-provisioned cleaner falls behind and leaves duplicate keys in place.

How Retention Affects Storage and Performance

Retention is the dominant driver of storage costs: ingest rate times retention time times replication factor gives the floor, so 100 MB/s for seven days at RF=3 is roughly 180 TB. Managing storage costs therefore starts with the window rather than with the hardware order. Longer windows do not slow reads of recent data, which is served from page cache, but they multiply open file handles and lengthen log recovery. Cleanup itself is a file unlink; compaction consumes real CPU and I/O.

Best Practices and Common Challenges

• Set retention per topic rather than one cluster-wide value; audit and financial topics rarely need the same window as debug telemetry, and per-topic windows are the only practical way to honor differing data retention policies.
• Pair the time and size-based retention limits: time protects compliance obligations, size protects the volume, and whichever triggers first wins. Expiring segments based on time alone leaves you exposed to a traffic spike.
• Align segment.ms with your shortest window so segments actually roll.
• Enable log compaction only where the topic really is a keyed changelog; on a plain event stream it is wasted CPU.
• Never treat a compacted topic as a backup: tombstones expire, and delete.retention.ms decides when.
• Change retention settings with kafka-configs.sh, not by editing server files, then watch log-cleaner metrics.

Shortening a window on a live topic is irreversible — one of the few mistakes that destroys data rather than merely delaying it. AceMQ's 11+ senior Kafka SMEs review these settings alongside capacity planning for 130+ enterprise customers across 26+ countries. If a cluster is filling up right now, 24/7 Kafka support carries a 15-minute emergency response SLA, alongside the full range of enterprise Kafka services.