Kafka vs Kinesis Compared

A Kinesis stream is a set of shards. Each record carries a partition key, which Kinesis hashes with MD5 to pick a shard, and gets a sequence number on write. In provisioned mode a shard accepts up to 1 MB/s or 1,000 records/s and serves 2 MB/s to readers, and you resize with UpdateShardCount or by splitting and merging shards. On-demand Standard removes that arithmetic: a new stream starts at 4 MB/s of write capacity and absorbs up to double its peak of the previous 30 days, though traffic that more than doubles within 15 minutes is throttled. On-demand Advantage, an account-level setting, drops the per-stream charge and allows pre-warming before an event, in exchange for committing to at least 25 MiB/s of ingest and of retrieval.

Retention is 24 hours by default and can be raised to 8,760 hours, 365 days. One limit catches teams out: on-demand mode splits busy shards but cannot isolate a hot partition key, so a single key above 1 MB/s or 1,000 records/s throttles in any mode.

A Kafka topic is split into partitions spread across brokers, each a replicated log on disk. Retention is per topic, seven days by default, and can be set by time, size or to unlimited. Log compaction, which Kinesis has no equivalent of, keeps the latest value for every key indefinitely, so a compacted topic can serve as the source of truth for changelogs and state rebuilds. Idempotent producers and transactions give exactly-once processing between topics.

Amazon MSK runs open-source Kafka in your AWS account, with Standard or Express brokers or MSK Serverless. Existing clients, connectors and tooling work unchanged. AWS supports the service; partitions, consumer groups and connectors remain yours, as managed Kafka options compared sets out.

Both order records only within a shard or partition, so the key decides what stays in sequence, and both deliver at least once. AWS documents duplicates from producer retries and from consumers restarting at the last checkpoint after a failure or rebalance, and advises a unique key in each record for deduplication. Kafka removes producer-side duplicates itself and can commit consumed offsets and produced records in one transaction.

Consumption is where they differ most. A Kinesis shard's 2 MB/s and five GetRecords calls per second are shared by every standard consumer, so AWS recommends enhanced fan-out for more than one consumer application: each registered consumer gets its own 2 MB/s per shard, up to the per-stream cap. KCL applications keep leases and checkpoints in a DynamoDB table, one more resource to size. Lambda reads each shard in order and stops on that shard when the function returns an error, so one bad record holds up everything behind it. Kafka consumer groups store offsets in Kafka, any number of groups can read a topic, and share groups, production-ready since Kafka 4.2, add queue-style consumption with per-record acknowledgement.

Kinesis integrates natively with Lambda, Amazon Data Firehose, Managed Service for Apache Flink, EventBridge and Glue; if every producer and consumer is an AWS service, that is the main reason to choose it. Kafka brings Kafka Connect, Kafka Streams, schema registries and clients in most languages, and its protocol is also spoken by MSK, Confluent Cloud, Azure Event Hubs and Redpanda. An application written against Kafka can move between them; one written against the Kinesis API stays on AWS.

Rates vary by region, so compare the meters on the Kinesis Data Streams pricing page rather than a quoted price. Provisioned bills shard-hours, PUT payload units counted in 25 KB chunks per record, retention beyond 24 hours, and enhanced fan-out per consumer-shard hour and per GB. On-demand Standard bills per GB written, each record rounded up to 1 KB, per GB read, per stream-hour, plus retention and fan-out. On-demand Advantage drops the stream-hour charge and the fan-out premium for its minimum commitment.

Kafka's bill follows infrastructure. MSK provisioned clusters bill broker hours and storage, plus a per-GB charge for data written to Express brokers, and do not charge for replication between brokers; self-managed Kafka on EC2 pays for traffic between availability zones and, usually the largest line, the engineers who run it. Small, spiky or single-consumer streams tend to favor Kinesis on-demand; steady high throughput, many consumers or long retention tend to favor Kafka.

1. Does anything run outside AWS, or might the workload move? Kafka.
2. Do you need retention beyond 365 days, compaction or exactly-once between topics? Kafka.
3. Will many independent applications read the same stream? Kafka, without a per-consumer cap.
4. Is it all-AWS, with Lambda, Firehose or Flink as consumers and nobody to run brokers? Kinesis.

Moving from Kinesis to Kafka is mostly a consumer rewrite, since DynamoDB checkpoints do not carry over: cut over at a known position and run both until outputs agree.