The payload got bigger and the whole worker went with it

Overview

Mule's default in-memory repeatable streaming works fine until a payload exceeds the configured buffer, at which point the behavior depends entirely on a streaming strategy most applications never set. Large-file and large-response integrations are where this surfaces, usually in production and usually after a partner grows.

Challenge

A partner file integration started failing when file sizes grew past a few hundred megabytes. The flow used DataWeave transformations and a For Each scope over the full payload, with the default in-memory repeatable stream, so the entire document was materialized in heap more than once. Workers were being killed mid-processing and the partner file was reprocessed from the beginning on retry, compounding the load.

Environment

Mule applications on CloudHub workers processing partner file feeds and API traffic, integrating with cloud databases and internal services.

Approach

AceMQ profiled worker heap during a reproduction of the failing file to confirm where the payload was being materialized, then reworked the flow to stream rather than buffer. Note that AceMQ does not operate the MuleSoft platform — this work is on the customer's applications, flows, and integration design running on it.

Solution

• Profiled worker heap during a reproduced failure to identify where the payload was materialized
• Configured file-store repeatable streaming with appropriate buffer sizing for large payloads
• Reworked DataWeave transformations to stream rather than load the full document into memory
• Replaced whole-payload iteration with batch processing and bounded record blocks
• Made retries resume at a record checkpoint instead of restarting the entire file
• Load-tested with payloads well above the largest observed partner file

Outcome

Technologies