Adding nodes had not fixed it
Apache Kafka
DockerLinuxProducer timeouts were addressed at their source rather than masked by additional capacity, and the team gained a clear signal set for detecting a recurrence early.
Overview
A hosting provider running roughly 2,000 servers for 60,000 customers had scaled its Kafka deployment from one node to five as demand grew, and was still seeing frequent producer timeouts. The instinct — add capacity — had already been tried.
Challenge
The cluster ran a replication factor of four across five nodes, with brokers and controllers colocated. That left the controller leader carrying an unusually high number of active connections on top of its replication duties, and it was the connection concentration rather than raw throughput that produced the timeouts. Re-elections and topic shifting followed, which read as instability rather than as a symptom of the same cause.
Environment
Five-node Kafka cluster deployed via Docker on physical servers, supporting a large shared hosting estate.
Approach
AceMQ reviewed the replication and topology choices against what the workload actually required, rather than tuning around the symptom. The recommendations were sequenced so each could be validated on cluster stability metrics before the next was applied.
Solution
- 1Reduced the replication factor from four to three, matching durability requirements without over-replicating every partition
- 2Separated broker and controller roles so controller work no longer competed with serving traffic
- 3Retained a minimum of three broker nodes to preserve availability guarantees through the change
- 4Tuned thread settings to improve responsiveness under connection load
- 5Established re-election frequency and topic shifting as the metrics to watch when validating each step
Outcome
Producer timeouts were addressed at their source rather than masked by additional capacity, and the team gained a clear signal set for detecting a recurrence early.
Technologies
Apache KafkaDockerLinuxRelated Use Cases
High-Throughput Kafka and RabbitMQ Support for Global Payments
AceMQ provided ongoing support for PagoNXT's high-throughput payments infrastructure running Kafka and RabbitMQ at 2,500 transactions per second, including load testing validation and architecture optimization.
Kafka CVE Patching and Compliance Strategy for Global Enterprise
AceMQ developed a multi-technology compliance strategy covering CVE patching for Kafka alongside RabbitMQ and IBM MQ deployments, creating a unified vulnerability management approach across the entire enterprise messaging stack.
Need Expert Apache Kafka Support?
AceMQ's senior Apache Kafka engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.