Overview
A payments processor ran ActiveMQ Classic as a master and slave pair behind its payment initiation, settlement and reporting services. A few times a week, producer threads in services that had nothing to do with settlement hung for minutes. The broker log carried the line that gives the mechanism away, a Usage Manager memory limit reached with producers being throttled, but nobody had connected it to a single destination. Restarting the settlement consumer cleared it, which made the problem easy to live with.
Challenge
Every destination drew on the broker's one shared memory limit, and the policy entries set no memoryLimit of their own. When a settlement-file consumer slowed behind a database lock, its queue grew and used the shared limit up. Producer flow control then throttled senders to every queue, including queues whose consumers were healthy. A default queue prefetch of 1000 on large messages made it worse, because messages dispatched to the slow consumer and not yet acknowledged still counted against broker memory.
Environment
ActiveMQ Classic 5.16.x as a master and slave pair on RHEL 8, KahaDB on a shared volume, about forty queues, Spring JMS producers and message listener container consumers connecting over OpenWire.
Approach
AceMQ worked in four live sessions with the customer's engineers rather than from a report. Each session captured broker JMX, thread dumps of blocked producers and the broker log, then changed one setting on a staging pair that reproduced the stall by pausing a consumer. Nothing reached production until the staging pair no longer blocked.
Solution
- Traced the stall to a single queue by comparing per-destination memory usage and queue size in JMX with the flow control lines in the broker log
- Added policy entries with a memoryLimit per destination class, so one backlog could no longer use the shared limit
- Moved large-message queues to a store cursor so a backlog pages to KahaDB instead of holding broker memory
- Cut prefetch on the slow queues to a value sized for the slowest consumer instead of the default
- Set a send timeout on producers for non-critical flows, so a blocked send fails fast and raises an alert rather than hanging a thread
- Added alerts on broker and per-queue memory and store percent usage, and wrote a runbook for finding the slow consumer from JMX
Outcome
A later slowdown in the settlement consumer backed up that queue alone. Its backlog paged to disk while payment initiation kept sending. The team now sees memory pressure on a destination before producers are affected, and can find the slow consumer from the runbook without escalating.
Technologies
ActiveMQ
Spring Framework
Java
Linux
Kubernetes
Prometheus
Grafana
