Without backpressure thresholds, NiFi will happily fill the disk

Overview

NiFi holds flowfile content on disk in the content repository, and connections without backpressure thresholds will queue indefinitely. When a downstream processor slows or stops, upstream processors keep producing and the repository grows until the volume is full — at which point the node cannot write provenance either and effectively stops. AceMQ recovers the node and sets the limits that should have been there.

Challenge

The failure sequence is consistent: a destination system becomes unavailable or slow, its processor stops draining its inbound queue, upstream processors continue producing, and connection queues grow without bound because the default thresholds were never adjusted. Disk fills, the provenance repository can no longer write, and the node stops processing entirely. Recovery is delicate because deleting repository files by hand corrupts flowfile state, and archived content complicates the picture.

Environment

Apache NiFi clusters on-premises or in cloud VMs with separate flowfile, content, and provenance repositories, ingesting from field and enterprise systems.

Approach

AceMQ recovers disk space safely — content archive expiration and controlled queue draining rather than manual file deletion — then restores the node and the flow. The durable work is setting backpressure object and data-size thresholds on every connection, adding load-balancing and prioritization where appropriate, and separating repositories onto their own volumes so one filling does not stop the others.

Solution

• Safe disk recovery through content archive expiration and controlled queue draining, without manual repository file deletion
• Backpressure object count and data size thresholds applied to every connection, sized against downstream throughput
• Content repository archive settings, retention period, and usage percentage tuned to the available volume
• Repository separation onto dedicated volumes so content growth cannot stop provenance and flowfile writes
• Failure-path design for the affected processors, routing to a retry or holding flow instead of queuing indefinitely
• Disk usage, queue depth, and backpressure alerting so the condition raises an alarm well before the volume fills

Outcome

Technologies