Adding nodes had not fixed it
Apache Kafka
DockerLinuxProducer timeouts were addressed at their source rather than masked by additional capacity, and the team gained a clear signal set for detecting a recurrence early.
Overview
A hosting provider running roughly 2,000 servers for 60,000 customers had scaled its Kafka deployment from one node to five as demand grew, and was still seeing frequent producer timeouts. The instinct — add capacity — had already been tried.
Challenge
The cluster ran a replication factor of four across five nodes, with brokers and controllers colocated. That left the controller leader carrying an unusually high number of active connections on top of its replication duties, and it was the connection concentration rather than raw throughput that produced the timeouts. Re-elections and topic shifting followed, which read as instability rather than as a symptom of the same cause.
Environment
Five-node Kafka cluster deployed via Docker on physical servers, supporting a large shared hosting estate.
Approach
AceMQ reviewed the replication and topology choices against what the workload actually required, rather than tuning around the symptom. The recommendations were sequenced so each could be validated on cluster stability metrics before the next was applied.
Solution
- 1Reduced the replication factor from four to three, matching durability requirements without over-replicating every partition
- 2Separated broker and controller roles so controller work no longer competed with serving traffic
- 3Retained a minimum of three broker nodes to preserve availability guarantees through the change
- 4Tuned thread settings to improve responsiveness under connection load
- 5Established re-election frequency and topic shifting as the metrics to watch when validating each step
Outcome
Producer timeouts were addressed at their source rather than masked by additional capacity, and the team gained a clear signal set for detecting a recurrence early.
Technologies
Apache KafkaDockerLinuxRelated Use Cases
High-Throughput Kafka and RabbitMQ Support for Global Payments
AceMQ provided ongoing support for PagoNXT's high-throughput payments infrastructure running Kafka and RabbitMQ at 2,500 transactions per second, including load testing validation and architecture optimization.
Kafka CVE Patching and Compliance Strategy for Global Enterprise
AceMQ developed a multi-technology compliance strategy covering CVE patching for Kafka alongside RabbitMQ and IBM MQ deployments, creating a unified vulnerability management approach across the entire enterprise messaging stack.
Memcached Cache Stampede and Eviction Support
Ongoing support for a Memcached tier prone to thundering-herd database load after node changes and cache expiry cliffs.
Docker Image Build and Layer Caching Optimization
Restructuring Dockerfiles and CI caching so builds reuse layers properly, cutting pipeline time and image size across a large service estate.
Docker OOMKilled Container Remediation
Resolving containers repeatedly OOMKilled because the JVM and Node runtimes inside them were sizing heap against host memory rather than the cgroup limit.
Recovering Transaction Throughput After a Kafka Rollout
Throughput fell from 320,000 to roughly 45,000 transactions per hour after Kafka was introduced. The platform needed 80,000 to prove it could scale.
Need Expert Apache Kafka Support?
AceMQ's senior Apache Kafka engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.