Is exactly once semantics possible in Kafka?

Yes, Apache Kafka has supported exactly-once semantics since version 0.11, released in 2017. It is implemented as exactly once processing rather than exactly once delivery over the network: records may still be transmitted more than once, but the broker discards the repeats and commits results atomically, so the outcome matches a single delivery. The guarantee applies to data written to Kafka topics and to the consumer positions committed in the same transaction. Any external system your application writes to must supply its own protection against repeats.

How do I enable exactly-once delivery in Kafka?

Configure three things together. On the producer, set enable.idempotence=true and a stable, unique transactional.id, then wrap sends in initTransactions(), beginTransaction(), and commitTransaction(). On the consumer, turn off auto-commit and set isolation.level=read_committed, then pass consumed positions to sendOffsetsToTransaction() so they commit atomically with your results. If you use Kafka Streams, processing.guarantee=exactly_once_v2 does all of this for you.

Is Kafka at-least-once by default?

Effectively yes. A modern Kafka producer defaults to enable.idempotence=true and acks=all, which suppresses repeats caused by producer retries, but consumers still commit their position separately from any processing they perform. A crash between handling a record and committing causes that record to be handled again on recovery, which is at-least-once delivery semantics. Reaching exactly once needs transactions on the producer and read_committed isolation on the consumer.

What is the consume-transform-produce pattern in Kafka?

Consume-transform-produce is the standard exactly once loop: an application reads records from an input topic, transforms them, writes results to an output topic, and commits the input position. The exactly once property comes from committing that position inside the same transaction as the results, using sendOffsetsToTransaction(). If the commit is rolled back, neither the written records nor the position advance become visible, so the work is repeated cleanly. Kafka Streams implements this pattern internally when exactly once is enabled.

What is the performance cost of using Kafka transactions?

Idempotence adds almost no measurable overhead and is enabled by default. Transactions add latency because records are only visible to read_committed consumers after commit markers are written, so throughput depends heavily on how much work each commit covers. Large batches amortise that cost well and typically give up a modest percentage of throughput, while very small ones can be dramatically slower. The larger operational risk is a stuck commit blocking downstream consumers until its timeout expires.

Kafka Exactly-Once Semantics: How It Works & Its Limits

# Kafka Exactly-Once Semantics: How It Works and Where It Stops

Kafka exactly-once semantics (EOS) is a set of guarantees that makes each record land in the log and advance its consumer position once and only once, even when producers retry, brokers fail, or an application restarts. Apache Kafka implements this with two mechanisms: an idempotent producer that discards a duplicate write using a unique producer ID and a per-partition sequence number, and transactions that turn writes to several topics plus the offset commit into one atomic unit. It is the strongest delivery guarantee the platform offers, and it is real — but it holds inside Kafka, not across an external database, object store, or third-party service you write to, which is where most teams get burned.

Is exactly-once semantics in Kafka actually possible?

Yes. Version 0.11 introduced exactly once semantics in 2017, and Confluent argued at the time that these message semantics are possible in a real messaging system, not only in theory. The caveat is definitional: no distributed system can promise a single network delivery happens once, because a sender cannot distinguish a lost request from a lost acknowledgement. What Kafka provides is *exactly-once processing* — failures still cause repeated attempts, but the log discards the repeats and commits their effects atomically, so the outcome matches a world where every message arrived once.

The three delivery guarantees, and which one is the default

Every messaging system offers three message delivery semantics.

At-most-once delivery. No producer retries; the consumer commits before processing. Fast, never duplicated, and any failure means data loss.
At-least-once semantics. The producer resends; the consumer commits after processing. Nothing is lost, but a duplicate is possible — the practical default.
Exactly once. Repeats are discarded and the commit is atomic with the output.

Since Kafka 3.0 the Java producer ships with enable.idempotence=true and acks=all, so a stock producer already suppresses retry duplicates. End to end that is still at least once: without transactions and a read_committed consumer, a crash between processing and committing replays records.

Why exactly once is a hard problem

Four failure modes must be solved at once, each classic in distributed systems. A producer that resends after a lost acknowledgement appends the same record twice. A consumer that dies after processing but before committing reprocesses on recovery. A task that writes results *and* commits its position moves two independent pieces of state that must land together or not at all. And a zombie instance, cut off but still running, keeps writing into a stream its replacement has taken over.

The building blocks Kafka uses

What Kafka guarantees rests on five components: the idempotence layer in the producer; a stable transactional.id; a broker-side transaction coordinator; the internal __transaction_state topic, plus commit and abort markers written into each data partition; and the consumer-side isolation.level=read_committed switch.

How do idempotent producers remove duplicates?

Set enable.idempotence=true and the broker assigns that producer a unique producer ID. Every batch carries the ID and a counter that rises monotonically within a topic. The broker remembers the highest value it accepted, so a resent batch arriving after a lost ack is recognised as a repeat, dropped, and acknowledged as success — one write, one record in the log.

That covers producer retries inside a single session only. Restart the process and it gets a new producer ID, so writes from before the crash cannot be matched — which is what transactional.id fixes.

How transactional producers and consumers work together

A transactional producer sets a stable, unique transactional.id, then follows a fixed call sequence: initTransactions() at startup, then beginTransaction(), send() calls, and commitTransaction() or abortTransaction().

initTransactions() registers with the transaction coordinator, bumps the producer epoch to fence any older instance holding the same identity, and resolves unfinished work — which makes recovery safe. On commit, the coordinator records the outcome in __transaction_state, then writes markers to every topic involved, giving atomic writes across multiple partitions.

Exactly once is a contract between producer and consumer, and none of it reaches the reading side unless you configure isolation.level=read_committed. The default, read_uncommitted, returns rolled-back records and defeats the design; a read_committed consumer skips aborted data and will not read past the last stable offset.

The consume-transform-produce pattern

Most exactly once processing in production is a consume-transform-produce loop: read an input Kafka topic, transform, write an output topic, commit the input position. The commit belongs *inside* the transaction. Turn off enable.auto.commit and hand the consumed offsets to sendOffsetsToTransaction(), so results and positions succeed or fail as one unit.

A stream processing application built on the Kafka Streams API gets this free: set processing.guarantee=exactly_once_v2 and the framework handles it, including the state stores it keeps for joins and aggregations over data streams. The v2 implementation uses one producer per instance, fenced through consumer group metadata, rather than the original per-task model — which is why exactly once is cheap enough to leave on in any streams app.

Where the guarantee stops

This is the part competing articles blur. A Kafka transaction covers the log and the consumer offsets committed with it. Nothing else. If your consumer writes to Postgres, S3, or a payments gateway, that write is not enrolled, and a crash in between produces a double write downstream even though the broker's own state is perfect. End-to-end exactly once therefore needs the sink to cooperate:

Make the sink idempotent. Upsert on a natural key, or write to a deterministic object path — the simplest and most durable answer.
Store positions in the sink. Commit data and offset in one external transaction, then seek from that value on recovery.
Use a connector that participates. Kafka Connect supports exactly-once delivery for source connectors and for sinks implementing two-phase commit — verify per connector.

Side effects outside the commit scope — sending an email, charging a card — can never be covered; de-dupe those at the boundary with an idempotency key. If a pipeline is emitting repeats you cannot explain, AceMQ's 24/7 Kafka support puts a senior engineer on it with a 15-minute emergency response SLA.

Performance implications and trade-offs

Idempotence is close to free and already on by default. Using Kafka transactions costs more: records become visible to read_committed consumers only once markers are written, so latency is roughly bounded by your commit interval — Kafka Streams tightens commit.interval.ms to 100ms under EOS for that reason. Overhead stays modest when a commit covers a healthy batch of transactional messages, and grows severe when it covers very few.

A hung transaction is the sharper risk: it pins the last stable offset and stalls every read_committed consumer on that partition until transaction.timeout.ms expires. Teams that use Kafka for ledgers or billing judge that worthwhile; for analytics, at least once plus a sink that upserts is simpler and cheaper.

How to tell whether exactly once is actually working

Configuration that looks correct in a properties file is not evidence. Four checks catch most of what goes wrong.

isolation.level on every consumer of the output topic. The default is read_uncommitted, so one unconverted downstream service quietly reads rolled-back records — the most common misconfiguration by a wide margin.
A stable transactional.id that survives restarts. An identity derived from a container hostname changes on every deployment and cannot fence its own zombie.
Lag on read_committed consumers. A partition whose lag stops falling while the log keeps growing usually means a transaction was never committed or aborted.
A producer killed mid-transaction in a test environment. Confirm nothing from the aborted attempt reaches the output topic; production is a poor place to exercise that path first.

Deciding to achieve exactly once is a design call; maintaining exactly-once semantics through upgrades, rebalances, and coordinator failovers is an operations problem. AceMQ's 11+ senior SMEs support 130+ enterprise customers across 26+ countries. Explore AceMQ's enterprise Kafka services to have that boundary drawn before it costs a reconciliation.