Overview
This is a typical engagement for a platform team that runs dozens of Java services on Kubernetes and has just moved base images to a newer JDK. A few services now restart every few hours with exit code 137, and the team's first reaction is to raise the memory limit. AceMQ's engineers join an incident bridge to find what is really using the memory.
Challenge
Exit code 137 means the process was killed from outside, so the JVM writes no OutOfMemoryError and no heap dump. Heap graphs look healthy, which sends the team looking at the wrong thing. Raising the limit hides the problem for a while and makes the cluster more expensive. Some services also got a different default heap, because older Java 8 builds ignored container limits and sized the heap from the host.
Environment
Spring-based Java services on a managed Kubernetes cluster, container memory limits set per deployment, a mix of Java 8 and 17 base images moving to 21, and Prometheus metrics for heap and container memory.
Approach
AceMQ compares container memory with JVM memory over time rather than watching the heap alone. A test replica runs with Native Memory Tracking on, and the engineers read where the non-heap memory goes: metaspace, thread stacks, code cache, direct buffers and native libraries. They then change heap sizing and caps together, and check GC and CPU ergonomics against the container's real CPU limit.
Solution
- Plot container working-set memory next to heap, metaspace and thread count to show the gap the heap graph hides
- Run a test replica with Native Memory Tracking and read the summary with jcmd to find the largest non-heap consumers
- Move heap sizing to a percentage of the container limit that leaves headroom for memory outside the heap
- Cap thread pools and direct buffer memory where one library grows them without limit
- Check which collector and how many GC and compiler threads the JVM chooses under the container's CPU quota
- Add alerts on container memory against limit, so the next change shows up before a restart
Outcome
The engagement delivers a written root-cause analysis, a heap and memory standard for the team's base images, the thread and buffer caps that were missing, and alerts that watch the container rather than only the heap. The team stops treating the memory limit as the knob to turn whenever a service restarts.
Technologies
Java
Kubernetes
Docker
Prometheus
Linux
Kibana
Kong
PostgreSQL
Redis
