Scaling RabbitMQ to support hundreds of thousands of connected devices for a global telecom leader
RabbitMQ
KubernetesQuorum Queues
Prometheus
GrafanaThe client eliminated weekly RabbitMQ crashes, achieved significantly improved consumer throughput, and has a clear scaling roadmap to support growing device counts. The platform now handles 300,000+ …
Overview
The company's IoT platform relies on RabbitMQ to handle messaging for over 300,000 connected devices with plans for significant scaling. Weekly out-of-memory crashes and performance bottlenecks were threatening platform stability and growth plans.
Challenge
RabbitMQ nodes were crashing weekly due to out-of-memory errors caused by improper vertical scaling approach, low prefetch counts (set to 1), and missing TTL configurations on queues. Producer blocking from slow consumers was creating cascading failures. The team was using classic queues without mirroring, leaving them vulnerable to data loss during node failures.
Environment
Kubernetes deployment, RabbitMQ handling 300,000+ device connections, Spring Boot consumers, plans for horizontal scaling with Prometheus/Grafana monitoring.
Approach
AceMQ conducted a multi-day intensive remediation with live environment review, real-time configuration changes, and hands-on performance optimization. The team analyzed metrics, reviewed topology, and provided detailed architectural recommendations for scaling.
Solution
- 1Recommended horizontal scaling strategy (more nodes) instead of vertical scaling
- 2Increased prefetch count from 1 to 20 for dramatically improved consumer throughput
- 3Configured lazy queues to reduce producer blocking by enabling direct-to-disk writes
- 4Implemented per-queue memory limits and TTL policies to prevent OOM crashes
- 5Raised high water mark from 0.7 to 0.9 for better memory utilization
- 6Designed migration path from classic queues to quorum queues for high availability
- 7Provided Excel-based server requirements calculator for capacity planning
Outcome
The client eliminated weekly RabbitMQ crashes, achieved significantly improved consumer throughput, and has a clear scaling roadmap to support growing device counts. The platform now handles 300,000+ devices without stability issues.
Technologies
RabbitMQKubernetesQuorum QueuesPrometheusGrafanaRelated Use Cases
RabbitMQ Performance Tuning
Throughput, latency, and resource utilization optimization including queue design, publisher confirms, replication settings, and concurrency tuning.
RabbitMQ Kubernetes Stabilization
Hardening RabbitMQ in Kubernetes environments with StatefulSet tuning, quorum queue optimization, storage isolation, and memory/network configuration.
Facing a RabbitMQ Production Issue?
AceMQ's senior RabbitMQ engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.