Kafka Event-Driven Architecture: Patterns, Design and Failure Modes

A Kafka event-driven architecture uses Kafka topics as the durable log between services: producers publish facts such as OrderPlaced, and any number of consumer groups react independently, at their own pace, and can replay history. Kafka supplies per-partition ordering, retention and replay. It does not decide what an event contains, how schemas evolve, how to avoid dual writes, or what happens to a message that keeps failing. Those decisions are what this guide covers.

For the broker internals underneath, see Kafka architecture.

Martin Fowler's 2017 article separates patterns often lumped together, and the one you build decides topic design and retention.

Kafka's default retention is seven days (log.retention.hours=168), so any topic that state is rebuilt from needs an explicit retention decision. Most systems mix the first two patterns; event sourcing means replaying every event through code that still understands the oldest schema.

Granularity. One topic per event type (orders.order-placed) is easy to secure and evolve. One topic per entity stream (orders.events, holding placed, paid and shipped) keeps every change to an order in one sequence. Kafka orders records only within a partition, so if a consumer needs placed before paid for the same order, both types must share a topic.

Keys. Use the entity or aggregate ID as the record key, so the default partitioner sends every event for one order to the same partition. A key with a few very hot values, such as one large tenant, overloads one partition and one consumer. Adding partitions later changes where keys hash, breaking per-key order across the change, so size for growth up front; see our Kafka partition strategy guide. Set replication and retention per topic rather than relying on broker defaults.

Events are a contract between teams that may never talk. Put a schema (Avro, Protobuf or JSON Schema) on every topic and let a registry reject incompatible changes at registration, not in a consumer's stack trace. Confluent Schema Registry (Confluent Community License) defaults to BACKWARD: consumers on the new schema can read old data, so you may delete fields and add optional ones, and you upgrade consumers first. Apicurio Registry is an Apache 2.0 alternative.

• Use BACKWARD_TRANSITIVE or FULL_TRANSITIVE on topics that get replayed: transitive modes check every earlier version, and a replay reads them all.
• Never relax a shared topic to NONE to force a breaking change; publish a new topic version.
• For several event types in one topic, switch from the default TopicNameStrategy to RecordNameStrategy or TopicRecordNameStrategy.

Give every event an envelope: a unique event ID, the type, when it happened, and a correlation ID.

On the producer side, current Kafka clients default to acks=all and enable.idempotence=true, and the producer documentation says ordering is preserved with idempotence enabled as long as max.in.flight.requests.per.connection stays at 5 or below. Check that older services have not overridden them:

acks=all
enable.idempotence=true
max.in.flight.requests.per.connection=5

The idempotent producer only removes duplicates from its own retries. A consumer that processes a batch and dies before committing offsets processes it again after the rebalance. Make every handler idempotent: record processed event IDs in the same transaction as the side effect, or upsert keyed on entity and version.

The most common data-loss bug in event-driven services is the dual write: commit to the database, then publish to Kafka. If the process dies between the two, the database and the event stream disagree.

The transactional outbox removes the second write. The service inserts the event into an outbox table in the same transaction as the business change; Debezium reads the database's change log and its outbox event router turns each row into a domain event:

transforms=outbox
transforms.outbox.type=io.debezium.transforms.outbox.EventRouter
# outbox columns: id, aggregatetype, aggregateid, type, payload
# default topic: outbox.event.${routedByValue} (the aggregatetype)
# default record key: aggregateid

Keying by aggregateid keeps each aggregate's events in order. Purge old outbox rows on a schedule, and deduplicate on the outbox id, because a connector restart can publish a row twice. Our Debezium support engineers run this pattern on PostgreSQL, SQL Server and Oracle.

Every consumer group receives every event; within a group each partition goes to one member, so partition count caps parallelism. Kafka 4.0 made the new consumer group protocol (KIP-848) generally available for faster rebalances, but clients default to group.protocol=classic and must opt in with group.protocol=consumer; see our Kafka rebalance guide.

Kafka has no dead-letter queue for ordinary consumers, so failure handling is your design:

1. Retry in place for transient errors, with backoff and a cap. Order holds but the partition stalls, and each poll loop must finish within max.poll.interval.ms (five minutes by default) or the consumer leaves the group.
2. Retry topics such as orders.retry.5m for errors that may clear later. The partition keeps flowing, but the retried event is now out of order for its key.
3. A dead-letter topic for events that will never succeed, with headers recording the error, source topic, partition and offset, an alert, and a replay path.

Spring for Apache Kafka and similar frameworks implement retry and dead-letter topics. Kafka Connect sink connectors use errors.tolerance=all with errors.deadletterqueue.topic.name. For queue-style work without per-key order, Kafka 4.2 made share groups production-ready, with per-record acknowledgement.

Kafka transactions give exactly-once processing to consume-transform-produce pipelines that stay inside Kafka: a Kafka Streams application with processing.guarantee=exactly_once_v2, or a transactional producer that commits consumed offsets and output records atomically, read downstream with isolation.level=read_committed (the consumer default is read_uncommitted). It does not cover side effects outside Kafka, such as a database write or a payment call, which still need idempotent handlers. Our Kafka exactly-once semantics guide explains how it works.

• Consumer lag per partition, not only per group: one stuck partition hides inside a healthy total. See Kafka consumer lag.
• End-to-end latency: event timestamp versus processing time, per consumer group.
• Dead-letter topics: alert on any new record.
• Rebalance frequency, producer errors and under-replicated partitions.
• Tracing: a correlation ID in record headers, so one transaction can be followed across services.

• Poison pill. One malformed event fails every retry and blocks its partition. Fix: bounded retries, then a dead-letter topic.
• Rebalance loops. A slow handler exceeds max.poll.interval.ms, its partitions move, and the cycle repeats. Fix: lower max.poll.records or faster handlers.
• Silent loss through retention. A consumer offline longer than the topic's retention returns to deleted events and, with auto.offset.reset at its default latest, skips ahead without an error. See Kafka retention policy.
• Hot partitions from skewed keys, breaking schema changes forced through, and dual writes without an outbox.

If some flows are really task queues with routing and priorities, see RabbitMQ vs Kafka.

AceMQ's Kafka support engineers review topic, key, schema and retry design and take the call when it breaks, on self-managed Kafka, Confluent Platform and Amazon MSK; Kafka managed services covers running the clusters for you.

• Martin Fowler: What do you mean by “Event-Driven”?
• Apache Kafka: Producer configs (acks, idempotence, ordering)
• Apache Kafka: Consumer configs (group.protocol, max.poll.interval.ms, auto.offset.reset)
• Apache Kafka: Broker configs (log.retention.hours)
• Apache Kafka: Kafka Connect user guide (error reporting and dead letter queue)
• Apache Kafka 4.0.0 release announcement (KIP-848)
• Apache Kafka 4.2.0 release announcement (share groups)
• Confluent: Schema evolution and compatibility
• Debezium: Outbox Event Router
• Apicurio Registry (Apache 2.0)