Data Streaming Platform: Choose the Parts, Build It, Run It

What a data streaming platform includes

Vendors use the phrase for a product. In practice it is a stack, and a team usually needs only some of it. These are the parts, what each one does and the decision inside each.

Which parts do you need: Kafka, Flink, Pulsar or Redpanda?

The honest first question is whether you need a platform at all. A warehouse that only needs yesterday's changes sooner is often better served by a managed CDC service or an ELT tool. Where a platform is justified, the options differ less on features than on who will run them.

Version and support dates that set your upgrade plan

A streaming platform has several components with separate release clocks. These are the published dates and rules as of 31 October 2026, from each project's or vendor's own pages. They change, so confirm them before you plan.

Sources: kafka.apache.org community downloads and the Kafka 4.0 upgrade notes, docs.confluent.io versions and interoperability, flink.apache.org downloads, pulsar.apache.org release policy and docs.redpanda.com, all checked on 31 October 2026.

Stream processing: Flink, Kafka Streams or ksqlDB

These three solve the same problem in different places. Flink is a framework and distributed engine for stateful computations over bounded and unbounded streams, with Java, Python and SQL interfaces, and it runs as its own cluster. Kafka Streams is a client library that you embed in a Java or Scala application, with no separate processing cluster. ksqlDB is a SQL layer on Kafka under the Confluent Community License.

Two details change designs more than feature lists do. First, exactly-once is not one setting. Flink's checkpoints give consistent state, and delivery to Kafka as an exactly-once sink uses transactions that are committed when a checkpoint completes, so records become visible to read-committed consumers only after the checkpoint. Kafka Streams offers exactly_once_v2, which needs brokers 2.5 or newer. Second, Confluent's own documentation now recommends Confluent Platform for Apache Flink as the stream processing engine for new workloads and says ksqlDB remains fully supported for existing applications. If you run ksqlDB today, plan for that direction.

Moving from batch ETL to streaming without a rewrite

A rewrite is rarely needed. The route we design is staged, and every stage can be stopped without loss.

• Capture — Read changes from the source database log with change data capture into the platform, leaving the batch job untouched.
• Run beside — Build one streaming consumer, such as a dashboard table, and compare its output with the batch result until they agree.
• Move consumers — Switch consumers over one at a time, keeping the batch path until the last one has gone.
• Retire — Turn the batch job off only after a full business cycle of matching output.

Run it yourself, buy a managed service, or have AceMQ run it

A managed service from a cloud or Kafka vendor runs the brokers. It does not choose your partition counts, own your consumers' lag, fix your connector configuration or tell you a Flink job's state has outgrown its checkpoints. That layer stays with you on every option, and it is where most incidents start.

• Open source, run by your team — Full control and the lowest licence cost. You own upgrades, KRaft and Java prerequisites, and the pager.
• Vendor managed service or distribution — Less to operate at the broker layer. Check the version policy, how you leave, and who answers when the problem is in your topics and consumers rather than in the service.
• AceMQ support or managed operations — AceMQ supports Apache Kafka wherever it runs, including managed services and Confluent Platform, or runs the platform for you on your infrastructure or ours.

What a streaming platform needs on call

These are the conditions we build alerts and runbook steps for, whoever holds the pager.

• Consumer lag — Lag that grows is a symptom. The runbook separates a slow consumer from a stuck rebalance from a partition skew.
• Partition and controller health — Under-replicated partitions, offline partitions and controller quorum state, so a lost broker is found before a producer times out.
• Disk and retention — Retention set against disk and traffic, with an alert before a broker fills.
• Flink checkpoints — Checkpoint duration and age, state size growth and backpressure, because a job that cannot checkpoint cannot restart cleanly.
• Connector state — Failed tasks, stalled offsets and source log retention for change data capture.

After go-live: four tiers of cover

Who this is not for, and what we do not promise

• Warehouse-only loads — If the only consumer is a warehouse, a managed CDC service or an ELT tool will cost less than a platform. We say so in the review.
• A single small topic — One low-volume queue between two services does not need Kafka, Flink or Pulsar. A message broker is simpler.
• Application event design — Event modelling, sagas and schema design for services is a separate engagement under event-driven architecture consulting.
• Exactly-once as a promise — We design for the guarantee each sink really provides and make consumers idempotent. We do not promise exactly-once across a whole pipeline.
• Throughput or latency figures in advance — They depend on hardware, partitioning, message size and consumers. We measure in your environment and do not quote a number beforehand.
• Writing your stream jobs or connectors — We design, configure, tune and fix. We do not write your Flink jobs, Kafka Streams applications or new connectors.