Overview
A retail bank ran IBM MQ 9.2 on Linux for payment and statement traffic. Several times a month, applications that had nothing to do with each other began receiving resource problem errors on persistent puts, the queue manager error log reported its log was full, and the incident cleared on its own about the time a nightly job finished.
Challenge
The recovery log is shared by every application connected to the queue manager. A statement batch job read a very large number of messages inside one unit of work and committed only at the end, so the oldest log extent it needed stayed pinned for the whole run. Other applications' persistent messages kept writing to the log until the primary and secondary files were used up, and from then on every persistent put on the queue manager failed, not just the batch job's. The team had seen the symptom as a general capacity problem and had asked for more disk.
Environment
IBM MQ 9.2 on RHEL with circular logging, a single queue manager for payment and statement flows, JMS and C clients, a Prometheus exporter for queue manager metrics.
Approach
AceMQ joined during an occurrence and read the queue manager error log and the connection list for units of work that had been open longest. The oldest unit of work belonged to one application connection, and the timeline matched the nightly schedule exactly. The log was then sized from measured transaction duration instead of by guesswork.
Solution
- Identified the connection holding the oldest unit of work from the queue manager's connection and unit-of-work log information, and matched it to the batch job
- Changed the batch job to commit every fixed number of messages, so no unit of work outlived a bounded amount of log
- Resized the log primary and secondary files from measured peak log use and the longest legitimate transaction
- Set MAXDEPTH on the statement queues so a backlog stops at a known size instead of growing without limit
- Added alerts on log file system use and on the age of the oldest unit of work, through the existing Prometheus exporter
- Wrote a runbook for finding the connection pinning the log and deciding whether to commit, roll back or stop it
Outcome
Persistent puts on payment queues no longer fail during the nightly run, and the log usage graph shows a flat line where it used to climb. The team can identify a long-running unit of work from the runbook in minutes instead of waiting for the job to end.
Technologies
IBM MQ
Linux
Java
Prometheus
RabbitMQ
Spring Framework
Kubernetes
