Overview
Docker build cache, dangling images, stopped containers, and orphaned volumes accumulate silently. On a busy build agent the filesystem fills within weeks, and the first visible symptom is usually something unrelated failing — a node marked NotReady, or a database refusing writes on the same volume.
Challenge
Build agents were failing every few weeks with no-space errors, and on two occasions a shared node filled up and evicted unrelated production pods. Cleanup was manual and inconsistent. BuildKit cache mounts and multi-stage build layers were accumulating fastest, and a set of orphaned volumes from removed containers had never been reclaimed.
Environment
CI build agents and Kubernetes nodes running containerized workloads, with frequent image builds across many services.
Approach
AceMQ measured what was actually consuming disk before changing anything — separating build cache, image layers, container writable layers, logs, and orphaned volumes — because the right retention policy differs for each. We then established automated pruning with retention that preserves cache value instead of discarding it wholesale.
Solution
- Broke down disk consumption by build cache, image layers, container layers, logs, and orphaned volumes
- Set automated pruning with age and size-based retention that keeps recent cache useful
- Configured BuildKit cache retention limits so cache mounts stop growing without bound
- Enabled container log rotation with size limits at the daemon level
- Separated build agent storage from workload storage so a build cannot fill a production node
- Added disk usage alerting with thresholds that fire well before the filesystem is full
Outcome
The recurring no-space failures stopped, and build cache hit rates stayed high because pruning is retention-based rather than a full wipe. Production pods no longer share a filesystem with the build cache.
Technologies
Docker
Kubernetes
Airbyte
PostgreSQL
Redis
Kong
Snowflake
dbt
Kafka
