Overview
A hosting provider running roughly 2,000 servers for 60,000 customers had scaled its Kafka deployment from one node to five as demand grew, and was still seeing frequent producer timeouts. The instinct — add capacity — had already been tried.
Challenge
The cluster ran a replication factor of four across five nodes, with brokers and controllers colocated. That left the controller leader carrying an unusually high number of active connections on top of its replication duties, and it was the connection concentration rather than raw throughput that produced the timeouts. Re-elections and topic shifting followed, which read as instability rather than as a symptom of the same cause.
Environment
Five-node Kafka cluster deployed via Docker on physical servers, supporting a large shared hosting estate.
Approach
AceMQ reviewed the replication and topology choices against what the workload actually required, rather than tuning around the symptom. The recommendations were sequenced so each could be validated on cluster stability metrics before the next was applied.
Solution
- Reduced the replication factor from four to three, matching durability requirements without over-replicating every partition
- Separated broker and controller roles so controller work no longer competed with serving traffic
- Retained a minimum of three broker nodes to preserve availability guarantees through the change
- Tuned thread settings to improve responsiveness under connection load
- Established re-election frequency and topic shifting as the metrics to watch when validating each step
Outcome
Producer timeouts were addressed at their source rather than masked by additional capacity, and the team gained a clear signal set for detecting a recurrence early.
Technologies
Apache Kafka
Docker
Linux
RabbitMQ
IBM MQ
Java
Kubernetes
Prometheus
