The payload got bigger and the whole worker went with it
MuleSoft
Kubernetes
Docker
PostgreSQLThe out-of-memory failures stopped and the integration now processes files several times larger than the size that originally broke it, on the same worker sizing.
Overview
Mule's default in-memory repeatable streaming works fine until a payload exceeds the configured buffer, at which point the behavior depends entirely on a streaming strategy most applications never set. Large-file and large-response integrations are where this surfaces, usually in production and usually after a partner grows.
Challenge
A partner file integration started failing when file sizes grew past a few hundred megabytes. The flow used DataWeave transformations and a For Each scope over the full payload, with the default in-memory repeatable stream, so the entire document was materialized in heap more than once. Workers were being killed mid-processing and the partner file was reprocessed from the beginning on retry, compounding the load.
Environment
Mule applications on CloudHub workers processing partner file feeds and API traffic, integrating with cloud databases and internal services.
Approach
AceMQ profiled worker heap during a reproduction of the failing file to confirm where the payload was being materialized, then reworked the flow to stream rather than buffer. Note that AceMQ does not operate the MuleSoft platform — this work is on the customer's applications, flows, and integration design running on it.
Solution
- 1Profiled worker heap during a reproduced failure to identify where the payload was materialized
- 2Configured file-store repeatable streaming with appropriate buffer sizing for large payloads
- 3Reworked DataWeave transformations to stream rather than load the full document into memory
- 4Replaced whole-payload iteration with batch processing and bounded record blocks
- 5Made retries resume at a record checkpoint instead of restarting the entire file
- 6Load-tested with payloads well above the largest observed partner file
Outcome
The out-of-memory failures stopped and the integration now processes files several times larger than the size that originally broke it, on the same worker sizing.
Technologies
MuleSoftKubernetesDockerPostgreSQLRelated Use Cases
MuleSoft CloudHub Worker Restart and Memory Pressure Support
Investigating recurring CloudHub worker restarts under memory pressure and the application-level causes behind them.
MuleSoft Integration Estate Assessment
Inventorying and evaluating a MuleSoft estate for reliability, error handling, and reuse before committing to either investment or migration.
Facing a MuleSoft Production Issue?
AceMQ's senior MuleSoft engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.