Overview
A benefits administrator ran an ActiveMQ Classic 5.17 broker that moved eligibility and enrolment events between its member portal and back-office systems. Queue depths were close to zero, yet the KahaDB data volume grew for months until the store usage limit was reached and producers began to block. The team had already doubled the volume once, which bought a few weeks.
Challenge
A durable subscription on an events topic belonged to a claims-audit application that had been retired a year before. The subscription was still registered, so the broker kept every message published to the topic for it. KahaDB deletes a journal data file only when no message in it is still needed, and one retained message is enough to hold a whole file. The retained messages were spread across the journal, so almost no files could be removed, and the store looked full of data that nobody was reading.
Environment
ActiveMQ Classic 5.17.x on a single Linux host with KahaDB on a dedicated data volume, durable topic subscriptions for several internal applications, Java and Spring JMS clients.
Approach
AceMQ first showed that the growth was retained messages and not slow checkpoints: journal file count against the oldest file that still held live messages, and the list of inactive durable subscribers from JMX. The same pass confirmed which subscriptions were still wanted with their owners before anything was removed.
Solution
- Listed inactive durable subscribers from JMX and matched each client ID to an application owner
- Removed the abandoned subscriptions with the broker's destroy-durable-subscriber operation once the owners confirmed
- Watched checkpoint and cleanup cycles release journal files until the store settled at its working size
- Set an offline durable subscriber timeout and task schedule so a subscription that stays inactive expires on its own
- Moved fan-out to applications that no longer needed message retention onto virtual topics, which use a queue per consumer
- Added alerts on store percent usage and on the count of inactive durable subscribers, and sized the store volume from the measured growth rate
Outcome
The journal shrank as cleanup cycles ran, store usage dropped well below the level where producers block, and the volume no longer needed to grow. An inactive subscriber now raises an alert, or expires on the timeout, long before it can hold the disk.
Technologies
ActiveMQ
Java
Linux
RabbitMQ
Kubernetes
Spring Framework
Docker
Redis Sentinel
Prometheus
