Workers recycling under memory pressure, and why the platform can't tell you which flow

Overview

When a CloudHub worker recycles under memory pressure, the platform reports that it happened. It does not tell you which flow, which object store, or which connector leak caused it. That analysis is application-side, and it is where AceMQ works — we tune and debug what customers run on MuleSoft, we do not operate MuleSoft itself.

Challenge

Workers were recycling several times a day, dropping in-flight messages. The application used an in-memory object store for deduplication keys with no expiry, held cached lookup data that grew unbounded, and had a connector configuration that opened a new connection per request without pooling. Each cause on its own was survivable; together they crossed the worker's memory ceiling daily.

Environment

Mule applications on CloudHub across several environments, integrating point-of-sale, inventory, and ERP systems.

Approach

AceMQ correlated restart timestamps with application behavior and worker metrics to narrow the search, then reviewed object store configuration, caching scopes, and connector pooling as the three most common application-side memory sinks. Named senior engineers worked the issue directly with 24/7 coverage and a 15-minute emergency response SLA.

Solution

• Correlated worker recycle events with flow activity and memory metrics to isolate the contributing flows
• Moved unbounded in-memory object stores to persistent stores with explicit TTLs
• Bounded cache scopes and set eviction policies on lookup data that had grown without limit
• Enabled connection pooling on connectors that were opening a connection per request
• Added message-level durability so an unexpected recycle no longer drops in-flight work
• Right-sized worker allocation against measured steady-state and peak memory

Outcome

Technologies