What Actually Causes a Kafka Producer Timeout, and How Do I Fix It?

org.apache.kafka.common.errors.TimeoutException: Expiring N record(s) is one of the most common Kafka production errors — and one of the most misdiagnosed. Teams frequently reach for producer-side configuration changes (bumping request.timeout.ms, increasing retries) when the actual cause is on the broker side, specifically in how the cluster's controller and broker roles are distributed.

This post walks through a real diagnostic case: a five-node physical Kafka cluster experiencing frequent producer timeouts, the root cause identified, and the exact fix applied.

Controller and broker role contention on the same nodes.

In a KRaft-mode cluster (Kafka's post-ZooKeeper architecture), every node can be configured to run in a combined broker+controller role, or these roles can be separated onto dedicated nodes. When all nodes run both roles — common in smaller or organically-grown clusters — the controller and broker processes compete for the same CPU and memory on every node, all the time.

In a real case, a five-node Kafka cluster configured with all nodes as combined broker+controller was experiencing producer timeouts traced specifically to the node acting as controller leader having an excessive number of active connections, which was creating broker-level contention. As AceMQ's Kafka SME diagnosed it directly: "Right now, since your brokers — having them, like you said, having the controllers and the brokers on the same physical box — they're competing all the time for memory and for CPU, and it causes latency on logging. It causes problems for the metadata that has to work with, because they're competing to update each other. Split those apart."

Check whether your replication factor is proportionate to your cluster size and actual durability needs — over-replication creates real, measurable overhead.

In the same diagnostic case, the cluster was running a replication factor of 4 on a 5-node physical cluster. The direct assessment: "The replication factor 4 is excessive for a 5-node physical cluster because it's a high level of redundancy, but it comes with a big price, especially with your topic distribution on the five nodes. I would definitely back that down to three, especially with the volume size you have."

Why replication factor matters for producer timeouts specifically: every produced message must be acknowledged by the required number of in-sync replicas (governed by your acks and min.insync.replicas settings) before the producer receives confirmation. A higher replication factor means more replicas need to write and acknowledge before that confirmation comes back — and if those replica writes are competing with controller-role CPU/memory contention on the same nodes, the combination compounds into producer-visible timeouts.

Rule of thumb: replication factor 3 is the standard durability baseline for production Kafka. Going to 4 or higher should be a deliberate decision based on demonstrated durability requirements — not a default.

Yes, if you're running combined roles and experiencing producer-side latency or timeout issues — this is one of the highest-leverage architectural changes available.

The direct recommendation from the same diagnostic case: "Right now, since it's production... maybe get another controller in and put the controller separately, and keep at least a 3-node on the broker side." This doesn't require a full cluster rebuild — it means adding dedicated controller node(s) and reconfiguring existing nodes to run broker-only roles, reducing the CPU/memory contention on every node in the cluster.

Practical migration path:

1. Provision a new node (or repurpose an existing one) as a dedicated controller
2. Reconfigure remaining nodes to broker-only role
3. Maintain a minimum of 3 broker nodes for production durability
4. Validate cluster health and re-election behavior before declaring the migration complete

Two additional tuning levers were part of the same real-world fix:

Increase the Kafka threaded I/O setting. The specific recommendation given was to take the current threaded setting (num.io.threads or the equivalent network/IO thread configuration) and double it — directly addressing throughput bottlenecks under load without requiring architectural changes.

Monitor for leader re-elections and partition reassignment churn. Frequent controller re-elections or unplanned leadership shifts stall producer traffic while the cluster reorganizes. As the diagnosis put it: "Look at these two here — this will give you a good feel for instability, how they're behaving, and if they're having to do re-elections, if they're having to shift things around all the time. Because when they shift, it stops everything — it slows those topics down." Watching your leader election rate and partition reassignment frequency as ongoing metrics (not just during an incident) gives you an early warning system for the same contention pattern recurring.

No — producer-side configuration absolutely can be the root cause, and it's worth ruling in or out early:

• request.timeout.ms set too aggressively low for your actual network and broker response characteristics
• max.in.flight.requests.per.connection combined with retries causing message reordering issues that compound into apparent timeouts
• batch.size and linger.ms misconfigured for your throughput pattern, causing the producer to hold messages longer than expected before sending
• Insufficient producer-side buffer memory (buffer.memory) causing BufferExhaustedException that gets misread as a timeout

The diagnostic discipline that matters here: check broker-side health and controller/broker separation first, since that's the more common root cause in real deployments, before spending time tuning producer client configuration that may not be the actual bottleneck.

If you're mid-upgrade or recently changed TLS configuration and started seeing timeouts, don't assume TLS is unrelated — but also don't assume it's the primary cause if you have unresolved controller/broker contention.

In the same case, the client had recently attempted an SSL certificate change that failed and required a rollback, and separately reported producer timeouts. The guidance given was explicit about sequencing: address the replication factor and controller/broker separation first — "don't look right now at the SSL... get the replication down to three, add the brokers and split them away, and then see how it's performing... I wouldn't try to do the TLS/SSL thing until you get to Kafka 4." Stabilize the cluster's core architecture before layering additional changes like certificate rotation on top of an already-strained cluster.

The diagnostic pattern here — replication factor, controller/broker topology, threading configuration, and re-election frequency — is approachable for a team with solid Kafka operational experience. But the reason this specific case escalated to expert review is common: the symptoms (producer timeouts) don't obviously point to the cause (controller/broker contention) without cluster-level diagnostic experience across many different Kafka topologies.

If you're seeing producer timeouts and have already ruled out obvious network issues, the fastest path to root cause is usually a structured review of your cluster's role topology, replication configuration, and re-election metrics — not incremental tuning of producer client settings one parameter at a time.

Experiencing Kafka producer timeouts in production? Contact AceMQ for a cluster architecture review — we'll identify whether it's controller contention, replication overhead, or something else entirely. Not sure where to start? Talk to an AceMQ engineer.

• Kafka Support
• Kafka Consumer Lag: Causes, Monitoring, and How to Reduce It
• Kafka Rebalance: What Triggers It and How to Stop the Storms
• AceMQ Home