Kafka stream processing: which engine should you use?

Use Kafka Streams when the processing is a set of transforms and aggregates owned by one Java or Scala service and Kafka is the only system involved. Use Apache Flink when state is large, when you join several sources or sinks, when you want SQL, or when processing should scale apart from the application. ksqlDB gives SQL on Kafka, but Confluent now recommends Flink for new workloads.

This guide compares the three on where they run, what they hold in state, what delivery guarantee they really give and how they are supported, as of 31 October 2026. If you are deciding on the whole platform around the processor, start with our page on data streaming platform services.

Sources are listed at the end. ksqlDB runs on Kafka Streams, which is why its processing settings carry the ksql.streams prefix.

Exactly-once is not one switch. In Kafka Streams you set processing.guarantee to exactly_once_v2; the default is at_least_once, and the older exactly_once and exactly_once_beta values are deprecated (Kafka Streams configuration).

In Flink, checkpoints give consistent state after a failure, and checkpointing is off until you enable it. Writing to Kafka exactly once uses the Kafka sink's EXACTLY_ONCE delivery guarantee, which writes records inside a Kafka transaction that is committed when a checkpoint completes. It needs a transactional ID prefix, and records become visible to read-committed consumers only after that checkpoint (Flink Kafka connector). Set the transaction timeout above your longest checkpoint plus restart time, or a slow checkpoint turns into an aborted transaction.

The practical lesson is the same for both: guarantees stop at the edge of Kafka. A database or HTTP sink that is not transactional or idempotent will see duplicates after a restart, so design those writes to be repeatable. The Kafka side of this is covered in Kafka exactly-once semantics.

Kafka orders records only within a partition, so any stateful operation on a key depends on that key always landing on the same partition. A join or a re-key that changes the key makes the engine repartition the data, which costs a round trip through Kafka and extra storage. Choose keys with that in mind, as set out in Kafka partition strategy.

Both Flink and Kafka Streams can window by event time, the time recorded in the record, rather than by when the engine happens to see it. Both also need a policy for records that arrive after their window has closed: Flink calls it allowed lateness and Kafka Streams a grace period. Decide it deliberately. The default that suits a dashboard is rarely right for a billing total.

A consumer reads records and leaves state, joins and recovery to you. Kafka Streams adds a DSL and a Processor API, splits work into tasks that follow topic partitions, and keeps each task's state in a local store with a changelog topic so a failed instance can be rebuilt elsewhere. Setting num.standby.replicas keeps warm copies and shortens that recovery (Kafka Streams architecture). The cost is that state lives next to your application instances, so a rebalance can mean waiting for a store to restore.

• Kafka Streams: nothing beyond Kafka and your deployment pipeline, but your application team owns state size, rebalance behaviour and upgrades of the library.
• Flink: a cluster or a managed service, checkpoint storage, and someone who watches checkpoint duration, state growth and backpressure. A job that cannot checkpoint cannot restart cleanly.
• ksqlDB: a Confluent-licensed server. The Confluent Platform 8.4 documentation says Confluent Platform for Apache Flink is the recommended stream processing engine for new workloads and that ksqlDB remains fully supported for existing applications, so plan for that direction if you run it.

Managed Flink services exist, including Confluent Cloud for Apache Flink and Amazon Managed Service for Apache Flink, which AWS renamed from Kinesis Data Analytics in August 2023. They run the engine. Job state, checkpoint settings and sink behaviour stay with your team, which is where AceMQ's Flink consulting and Kafka consulting start.

1. Is the logic a few transforms inside one service? Start with Kafka Streams.
2. Does it join several sources, hold large state or need SQL for analysts? Start with Flink.
3. Do you already run ksqlDB? Keep it for existing queries, and put new work through the Flink question above.
4. Who is paged when it stalls? If nobody, solve that first. AceMQ offers Kafka support and Flink support around the clock, and the choice of processor does not change who answers.

• Apache Kafka: Kafka Streams core concepts, checked 31 October 2026
• Apache Kafka: Kafka Streams configuration, checked 31 October 2026
• Apache Kafka: Kafka Streams architecture, checked 31 October 2026
• Apache Flink: Kafka connector and delivery guarantees, checked 31 October 2026
• Apache Flink: stateful stream processing, checked 31 October 2026
• Confluent: ksqlDB overview, Confluent Platform 8.4, checked 31 October 2026
• Confluent: ksqlDB server configuration parameters, checked 31 October 2026
• confluentinc/ksql repository README, checked 31 October 2026
• AWS: Amazon Managed Service for Apache Flink renamed from Kinesis Data Analytics, 30 August 2023