The disk fills, the node goes NotReady, and nobody pruned anything
Docker
Kubernetes
Airbyte
PostgreSQLThe recurring no-space failures stopped, and build cache hit rates stayed high because pruning is retention-based rather than a full wipe. Production pods no longer share a filesystem with the build c…
Overview
Docker build cache, dangling images, stopped containers, and orphaned volumes accumulate silently. On a busy build agent the filesystem fills within weeks, and the first visible symptom is usually something unrelated failing — a node marked NotReady, or a database refusing writes on the same volume.
Challenge
Build agents were failing every few weeks with no-space errors, and on two occasions a shared node filled up and evicted unrelated production pods. Cleanup was manual and inconsistent. BuildKit cache mounts and multi-stage build layers were accumulating fastest, and a set of orphaned volumes from removed containers had never been reclaimed.
Environment
CI build agents and Kubernetes nodes running containerized workloads, with frequent image builds across many services.
Approach
AceMQ measured what was actually consuming disk before changing anything — separating build cache, image layers, container writable layers, logs, and orphaned volumes — because the right retention policy differs for each. We then established automated pruning with retention that preserves cache value instead of discarding it wholesale.
Solution
- 1Broke down disk consumption by build cache, image layers, container layers, logs, and orphaned volumes
- 2Set automated pruning with age and size-based retention that keeps recent cache useful
- 3Configured BuildKit cache retention limits so cache mounts stop growing without bound
- 4Enabled container log rotation with size limits at the daemon level
- 5Separated build agent storage from workload storage so a build cannot fill a production node
- 6Added disk usage alerting with thresholds that fire well before the filesystem is full
Outcome
The recurring no-space failures stopped, and build cache hit rates stayed high because pruning is retention-based rather than a full wipe. Production pods no longer share a filesystem with the build cache.
Technologies
DockerKubernetesAirbytePostgreSQLRelated Use Cases
Docker OOMKilled Container Remediation
Resolving containers repeatedly OOMKilled because the JVM and Node runtimes inside them were sizing heap against host memory rather than the cgroup limit.
Docker Image Build and Layer Caching Optimization
Restructuring Dockerfiles and CI caching so builds reuse layers properly, cutting pipeline time and image size across a large service estate.
Need Expert Docker Support?
AceMQ's senior Docker engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.