Kafka Performance Tuning Guide, 2026: Optimize Producers, Brokers and Consumers

Tuning without a baseline is guessing, so record producer throughput and latency, broker idle time and consumer lag under a repeatable load before changing a setting. Kafka ships the tools: kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh, plus JMX metrics on every client and broker. Export those metrics to Prometheus or whatever monitor Kafka already feeds, so the baseline and every later run are kept.

bin/kafka-producer-perf-test.sh --topic perf-test --num-records 5000000 \
  --record-size 1024 --throughput -1 \
  --producer-props bootstrap.servers=host:9092 acks=all linger.ms=5

bin/kafka-consumer-perf-test.sh --bootstrap-server host:9092 \
  --topic perf-test --messages 5000000

For a real-time pipeline, also measure end-to-end latency from produce to consume, not only producer latency. Use a test topic with the same partition count and replication factor as production. Kafka 4.4 adds --record-key-range, --key-distribution and --random-seed options to the producer tool, so a benchmark can reproduce a skewed key distribution instead of spreading records evenly.

Watch these while the test runs. Apache's monitoring documentation gives ideal values for the first two: RequestHandlerAvgIdlePercent and NetworkProcessorAvgIdlePercent above 0.3, under-replicated partitions at 0. On the clients, record-queue-time-avg, batch-size-avg and request-latency-avg for producers, and records-lag-max, fetch-latency-avg and records-consumed-rate for consumers.

Producer throughput is mostly batching: larger batches mean fewer requests, better compression and less broker work, paid for with a little latency. Four settings in the producer configuration control it for Kafka producers, and the defaults below are from the Apache Kafka 4.4 producer documentation.

Acknowledgements are a durability choice, not a tuning knob. acks=all is the default in 4.4 and the setting that, with min.insync.replicas=2 on a replication factor of 3, keeps acknowledged writes after a broker loss. Dropping to acks=1 buys latency by accepting that the leader can fail before followers copy the record. Change it only with the data owner's agreement. Leave idempotence on: it needs acks=all and at most 5 in-flight requests per connection, which is the default.

Retries are on by default (retries is 2147483647, bounded in practice by delivery.timeout.ms, with retry.backoff.ms at 100), so a slow broker shows up as latency and growing queue time rather than as errors, until the delivery timeout expires. Change one producer setting at a time and re-run the same perf test. Compression in particular needs a test on your own records, because JSON, Avro and already-compressed payloads compress very differently.

The partition count is the ceiling on parallel consumers and the main lever on throughput, and it can be raised but never lowered, so size it from measurement. Measure what one partition sustains for your producers (P) and for your consumers (C) on your hardware, then the partitions needed for a target throughput T are at least T divided by P and at least T divided by C. It also has to be at least the number of consumer instances you will run at peak. Take the largest of the three and add headroom.

Keys decide whether that arithmetic holds. All records with one key land on one partition, so a skewed key concentrates load there regardless of how many partitions exist. Compare per-partition bytes in and per-partition lag: one partition far above the median is a key problem, and the fix is a better key, a salted key where ordering allows it, or a custom partitioner. If the hot partition also sits on an overloaded broker, kafka-reassign-partitions.sh can move it, with a throttle so the copy does not starve production traffic, but that fixes placement and not skew. Adding partitions to a keyed topic also changes which partition each key maps to, which breaks per-key ordering across the change.

From our engagements. AceMQ's Kafka engagements include performance tuning and cutting oversized clusters, across self-managed Kafka, Amazon MSK and Confluent Platform. The ceiling a client hits is almost always a design ceiling, one hot topic or one oversized topic, not the broker's, and the fix is usually re-partitioning the workload rather than switching brokers.

Broker capacity is arithmetic on four inputs: producer ingress, replication factor, the number of consumer groups reading everything, and retention. Replication multiplies every byte, and each consumer group multiplies the outbound traffic, which is why network bandwidth and disk run out before CPU does on most Kafka cluster designs. Compare the result with each broker's network bandwidth and disk throughput, and set alert thresholds from it.

With cluster ingress I (MB/s), B brokers, replication factor r and c consumer groups that each read the full stream, and leaders spread evenly:

• Inbound per broker: r × I / B, because each broker leads I / B and copies the other (r - 1) shares as a follower.
• Outbound per broker: (r - 1 + c) × I / B, replication to followers plus consumer reads.
• Disk, whole cluster: I × 86,400 seconds × retention days × r, divided by your compression ratio.
• Page cache per broker: Apache's rule of thumb is write throughput multiplied by 30 seconds of buffering, so each broker's inbound rate times 30.

The inputs are illustrative, not a recommendation; replace them with your own. The result is a floor: a lagging consumer reads from disk instead of the page cache, so spare memory protects the whole cluster from one slow reader.

Broker threads and fetchers. num.network.threads (default 3), num.io.threads (default 8) and num.replica.fetchers (default 1) are the settings to raise when the idle percentages drop below 0.3 or followers fall out of the in-sync set. Raising replica fetchers adds parallelism between brokers at the cost of CPU and memory. If followers lag, check replica.lag.time.max.ms (default 30 seconds) before assuming the disk is slow.

JVM and scaling. Keep the JVM heap modest: Apache's example is -Xmx6g -Xms6g -XX:+UseG1GC -XX:MaxGCPauseMillis=20, which leaves the rest of RAM to the page cache. Scaling out by adding brokers does not move existing partitions by itself, so plan a reassignment before expecting a new broker to take load.

Disks and the operating system. Give Kafka its own drives, keep file descriptors at 100,000 or more and check vm.max_map_count against the partition count, as the production readiness checklist sets out in checks 17 to 20.

Consumer lag, the gap between a partition's log end offset and the group's committed offset, has a handful of causes, and the pattern of lag across partitions and over time points at which one you have. In a microservices estate each service is usually its own consumer group, so read lag per group. Check the pattern first: bin/kafka-consumer-groups.sh --bootstrap-server host:9092 --describe --group my-group shows lag per partition, and --members shows which instance owns each.

Adding consumers helps only until instances equal partitions. Past that, extra consumers sit idle, which is the usual finding behind a team that scaled the deployment and saw no change. Share groups, production-ready from Kafka 4.2, let consumers outnumber partitions for independent tasks, but they give up per-partition ordering.

Four 4.x changes alter tuning advice, and each is documented by Apache. The producer default linger.ms moved from 0 to 5 in 4.0, because larger batches usually give similar or lower latency. The next-generation consumer rebalance protocol (KIP-848) is generally available from 4.0 and is enabled on the client with group.protocol=consumer; it is fully incremental, so it has no global synchronisation barrier, and it moves heartbeat, session timeout and assignment to the broker, so session.timeout.ms, heartbeat.interval.ms and partition.assignment.strategy stop applying. Brokers need Java 17. And 4.4, released on 22 October 2026, adds key-distribution options to the producer perf tool.

Check your version against the supported set before tuning: on 28 October 2026 Apache maintains 4.4, 4.3 and 4.2. A cluster on a 3.x release can only reach them through a KRaft migration, which is why tuning and migration planning often get scoped together.

Tuning stops helping when the limit is design: a hot key, a topic that carries unrelated workloads, a partition count chosen for last year's traffic, or a consumer doing a blocking call per record. Those need a redesign, and the signs are a benchmark that barely moves after three settings changes and lag that follows one partition.

AceMQ works with 160+ customers and handles 40+ Sev 1 incidents a year across the platforms it supports. For Kafka, the usual entry is the Kafka health check, which covers partition skew, consumer behaviour, replication and capacity and returns ranked findings, with 24/7 support and a 15-minute emergency SLA for incidents. Where AceMQ is not the right call: a one-off benchmark on a cluster you already understand needs a day with the tools above, not a contract, and a fully managed service such as Confluent Cloud or Amazon MSK takes broker and disk tuning off your plate, though it leaves partitioning, keys and consumers with you.