Overview
Container hardening has a short list of high-value items: do not run as root, do not allow privilege escalation, make the root filesystem read-only, drop capabilities, and keep secrets out of image layers. The last one matters most because a deleted file in a later layer is still recoverable from the earlier one.
Challenge
Nearly every image ran as root by default, none set a read-only root filesystem, and several ran with added capabilities that were copied from an old troubleshooting session and never removed. Layer history analysis found credentials in build-time ARG values and a config file that had been added and then deleted in a later layer — still fully recoverable from the image.
Environment
Containerized applications on Kubernetes under security accreditation requirements, built from internally maintained base images.
Approach
AceMQ analyzed the image layer history for every service alongside its runtime security context, prioritizing findings by exploitability rather than by count. Secrets found in layers were treated as compromised and rotated, not merely removed from future builds.
Solution
- Scanned image layer history for credentials, keys, and config files removed in later layers
- Rotated every credential found in image history rather than only stripping it from future builds
- Converted images to run as a non-root user with an explicit UID and matching filesystem ownership
- Set read-only root filesystems with explicit writable mounts only where genuinely required
- Dropped all capabilities by default and re-added only those a service demonstrably needs
- Moved build-time secrets to BuildKit secret mounts so they never enter a layer
Outcome
Images run as non-root with read-only filesystems across the estate, exposed credentials were rotated, and the accreditation findings tied to container configuration were closed.
Technologies
Docker
Kubernetes
Kong
PostgreSQL
Redis
NGINX
Kafka
