The disk fills, the node goes NotReady, and nobody pruned anything

Overview

Docker build cache, dangling images, stopped containers, and orphaned volumes accumulate silently. On a busy build agent the filesystem fills within weeks, and the first visible symptom is usually something unrelated failing — a node marked NotReady, or a database refusing writes on the same volume.

Challenge

Build agents were failing every few weeks with no-space errors, and on two occasions a shared node filled up and evicted unrelated production pods. Cleanup was manual and inconsistent. BuildKit cache mounts and multi-stage build layers were accumulating fastest, and a set of orphaned volumes from removed containers had never been reclaimed.

Environment

CI build agents and Kubernetes nodes running containerized workloads, with frequent image builds across many services.

Approach

AceMQ measured what was actually consuming disk before changing anything — separating build cache, image layers, container writable layers, logs, and orphaned volumes — because the right retention policy differs for each. We then established automated pruning with retention that preserves cache value instead of discarding it wholesale.

Solution

• Broke down disk consumption by build cache, image layers, container layers, logs, and orphaned volumes
• Set automated pruning with age and size-based retention that keeps recent cache useful
• Configured BuildKit cache retention limits so cache mounts stop growing without bound
• Enabled container log rotation with size limits at the daemon level
• Separated build agent storage from workload storage so a build cannot fill a production node
• Added disk usage alerting with thresholds that fire well before the filesystem is full

Outcome

Technologies