The disk fills, the node goes NotReady, and nobody pruned anything
Docker
Kubernetes
Airbyte
PostgreSQLThe recurring no-space failures stopped, and build cache hit rates stayed high because pruning is retention-based rather than a full wipe. Production pods no longer share a filesystem with the build…
Overview
Docker build cache, dangling images, stopped containers, and orphaned volumes accumulate silently. On a busy build agent the filesystem fills within weeks, and the first visible symptom is usually something unrelated failing — a node marked NotReady, or a database refusing writes on the same volume.
Challenge
Build agents were failing every few weeks with no-space errors, and on two occasions a shared node filled up and evicted unrelated production pods. Cleanup was manual and inconsistent. BuildKit cache mounts and multi-stage build layers were accumulating fastest, and a set of orphaned volumes from removed containers had never been reclaimed.
Environment
CI build agents and Kubernetes nodes running containerized workloads, with frequent image builds across many services.
Approach
AceMQ measured what was actually consuming disk before changing anything — separating build cache, image layers, container writable layers, logs, and orphaned volumes — because the right retention policy differs for each. We then established automated pruning with retention that preserves cache value instead of discarding it wholesale.
Solution
- 1Broke down disk consumption by build cache, image layers, container layers, logs, and orphaned volumes
- 2Set automated pruning with age and size-based retention that keeps recent cache useful
- 3Configured BuildKit cache retention limits so cache mounts stop growing without bound
- 4Enabled container log rotation with size limits at the daemon level
- 5Separated build agent storage from workload storage so a build cannot fill a production node
- 6Added disk usage alerting with thresholds that fire well before the filesystem is full
Outcome
The recurring no-space failures stopped, and build cache hit rates stayed high because pruning is retention-based rather than a full wipe. Production pods no longer share a filesystem with the build cache.
Technologies
DockerKubernetesAirbytePostgreSQLRelated Use Cases
Docker OOMKilled Container Remediation
Resolving containers repeatedly OOMKilled because the JVM and Node runtimes inside them were sizing heap against host memory rather than the cgroup limit.
Docker Image Build and Layer Caching Optimization
Restructuring Dockerfiles and CI caching so builds reuse layers properly, cutting pipeline time and image size across a large service estate.
Airbyte Deployment Hardening and Operations Consulting
Taking a proof-of-concept Airbyte install to a production-grade deployment with proper isolation, secrets handling, resource limits, and recovery procedures.
Airbyte Connector Schema Drift Remediation
Diagnosing an Airbyte connection that kept reporting success while the destination table quietly went stale after an upstream schema change.
Airbyte Pipeline and Connector Estate Assessment
A structured review of an Airbyte estate that had grown organically — auditing connector versions, sync modes, state handling, and failure visibility.
WSO2 Platform Architecture Assessment
Reviewing a WSO2 deployment's topology, database layout, high availability posture, and gateway sizing against its actual traffic profile.
Need Expert Docker Support?
AceMQ's senior Docker engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.