Root containers, writable filesystems, and credentials in layer history

Overview

Container hardening has a short list of high-value items: do not run as root, do not allow privilege escalation, make the root filesystem read-only, drop capabilities, and keep secrets out of image layers. The last one matters most because a deleted file in a later layer is still recoverable from the earlier one.

Challenge

Nearly every image ran as root by default, none set a read-only root filesystem, and several ran with added capabilities that were copied from an old troubleshooting session and never removed. Layer history analysis found credentials in build-time ARG values and a config file that had been added and then deleted in a later layer — still fully recoverable from the image.

Environment

Containerized applications on Kubernetes under security accreditation requirements, built from internally maintained base images.

Approach

AceMQ analyzed the image layer history for every service alongside its runtime security context, prioritizing findings by exploitability rather than by count. Secrets found in layers were treated as compromised and rotated, not merely removed from future builds.

Solution

• Scanned image layer history for credentials, keys, and config files removed in later layers
• Rotated every credential found in image history rather than only stripping it from future builds
• Converted images to run as a non-root user with an explicit UID and matching filesystem ownership
• Set read-only root filesystems with explicit writable mounts only where genuinely required
• Dropped all capabilities by default and re-added only those a service demonstrably needs
• Moved build-time secrets to BuildKit secret mounts so they never enter a layer

Outcome

Technologies