Without backpressure thresholds, NiFi will happily fill the disk
Apache KafkaApache HadoopThe affected nodes return to service without flowfile loss, and connection-level backpressure means a downstream outage now slows the flow instead of filling the disk. Alerting catches the condition w…
Overview
NiFi holds flowfile content on disk in the content repository, and connections without backpressure thresholds will queue indefinitely. When a downstream processor slows or stops, upstream processors keep producing and the repository grows until the volume is full — at which point the node cannot write provenance either and effectively stops. AceMQ recovers the node and sets the limits that should have been there.
Challenge
The failure sequence is consistent: a destination system becomes unavailable or slow, its processor stops draining its inbound queue, upstream processors continue producing, and connection queues grow without bound because the default thresholds were never adjusted. Disk fills, the provenance repository can no longer write, and the node stops processing entirely. Recovery is delicate because deleting repository files by hand corrupts flowfile state, and archived content complicates the picture.
Environment
Apache NiFi clusters on-premises or in cloud VMs with separate flowfile, content, and provenance repositories, ingesting from field and enterprise systems.
Approach
AceMQ recovers disk space safely — content archive expiration and controlled queue draining rather than manual file deletion — then restores the node and the flow. The durable work is setting backpressure object and data-size thresholds on every connection, adding load-balancing and prioritization where appropriate, and separating repositories onto their own volumes so one filling does not stop the others.
Solution
- 1Safe disk recovery through content archive expiration and controlled queue draining, without manual repository file deletion
- 2Backpressure object count and data size thresholds applied to every connection, sized against downstream throughput
- 3Content repository archive settings, retention period, and usage percentage tuned to the available volume
- 4Repository separation onto dedicated volumes so content growth cannot stop provenance and flowfile writes
- 5Failure-path design for the affected processors, routing to a retry or holding flow instead of queuing indefinitely
- 6Disk usage, queue depth, and backpressure alerting so the condition raises an alarm well before the volume fills
Outcome
The affected nodes return to service without flowfile loss, and connection-level backpressure means a downstream outage now slows the flow instead of filling the disk. Alerting catches the condition while there is still room to act.
Technologies
Apache KafkaApache HadoopRelated Use Cases
Apache NiFi Cluster Node Disconnect Support
Resolving nodes disconnecting from a NiFi cluster under load due to heartbeat timeouts, GC pauses, and ZooKeeper coordination failures.
Apache NiFi Flow Design and Modernization
Restructuring sprawling NiFi canvases into versioned, parameterized, testable flows with a promotion path across environments.
Facing a Apache NiFi Production Issue?
AceMQ's senior Apache NiFi engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.