Most slow Docker builds are cache invalidation, not compute

Overview

Slow container builds are almost always a layer ordering problem. A COPY of the full source tree before dependency installation invalidates the dependency layer on every commit, and no amount of faster build hardware fixes that.

Challenge

The CI pipeline had grown past twenty minutes per service and developers were batching changes to avoid waiting. Dockerfiles copied the entire repository before installing dependencies, had no .dockerignore so build context included the git history and node_modules, ran single-stage builds that shipped compilers and build tooling into the runtime image, and pinned nothing, so builds were not reproducible.

Environment

Dozens of containerized services built in CI and deployed to Kubernetes, with images pulled across regions.

Approach

AceMQ restructured the Dockerfiles around cache stability — dependency manifests copied and installed before source, multi-stage builds separating build tooling from runtime — and enabled a shared registry-backed build cache so CI agents benefit from each other's work rather than starting cold.

Solution

• Reordered layers so dependency installation precedes source copy and survives ordinary commits
• Added .dockerignore files to cut build context size dramatically
• Converted single-stage builds to multi-stage, keeping compilers and build tooling out of runtime images
• Enabled registry-backed BuildKit cache export and import so CI agents share cache
• Pinned base images by digest and pinned dependency versions for reproducible builds
• Established a per-service image size budget checked in CI

Outcome

Technologies