The runtime sized itself for the host, then the cgroup killed it

Overview

OOMKilled with no application-level out-of-memory error usually means the runtime inside the container never learned about the container's memory limit. A JVM on an old base image, or a Node process with a manually set max-old-space-size copied from a bigger host, will size itself against total host memory and get killed by the kernel long before it thinks it is under pressure.

Challenge

Several services were being OOMKilled multiple times per day with no heap dump and no application error — the kernel killed the process before the runtime could report anything. Some services ran a JVM on a base image predating reliable container awareness, and others ran Node with a heap ceiling set higher than the container's own limit. Raising the memory limits had been tried and simply moved the threshold.

Environment

Containerized JVM and Node services on Kubernetes, built from a mix of base images maintained by different teams.

Approach

AceMQ mapped each affected container's runtime memory configuration against its cgroup limit to identify which processes were unaware of the limit and which were simply misconfigured. Both classes were fixed at the image and manifest level rather than by raising limits until the killing stopped.

Solution

• Compared each runtime's effective memory configuration against the container's actual cgroup limit
• Updated base images to runtime versions that honor cgroup limits, with explicit percentage-based heap settings
• Removed hardcoded heap ceilings that exceeded the container memory limit
• Accounted for off-heap, metaspace, and thread stack memory in the limit rather than heap alone
• Added container-level memory metrics and OOMKill alerting distinct from application error alerting
• Load-tested each service to establish a limit from measured peak rather than from trial and error

Outcome

Technologies