Overview
This is a typical engagement for a platform team that runs a few dozen Go services on Kubernetes with CPU limits set. Some show p99 latency spikes that do not follow traffic, and a few restart with exit code 137 under load. The services were built over several years with different Go versions. AceMQ and a partner Go specialist find where the time and memory go, using the service's own profiles.
Challenge
Go 1.25 changed the default GOMAXPROCS on Linux to respect the container's CPU limit, and Kubernetes CPU requests are ignored by that logic. A service built with an older Go, or with GOMAXPROCS set by hand, can run more threads than its quota allows and spend its time throttled. Exit code 137 means the kernel killed the process, so no Go panic appears, and heap graphs alone do not explain it.
Environment
Go services on a managed Kubernetes cluster, built with Go versions from several releases, CPU and memory limits set per deployment, Prometheus metrics for container CPU throttling and memory, and net/http/pprof endpoints on some services.
Approach
AceMQ lists the Go version each binary was built with (go version -m reads it from the binary), the GOMAXPROCS and GOMEMLIMIT settings in each deployment and the container limits. It then compares CPU throttling and working-set memory over time with Go's own runtime metrics, and takes CPU, heap and goroutine profiles from a canary replica rather than the whole fleet.
Solution
- Reads the Go version from each binary and the GOMAXPROCS and GOMEMLIMIT settings from each deployment
- Plots container CPU throttling next to request latency to show which services are limited by their quota
- Removes hand-set GOMAXPROCS values where the Go version already follows the cgroup limit, and sets it deliberately where it does not
- Sets GOMEMLIMIT below the container memory limit with headroom for memory outside the Go heap
- Takes CPU, heap and goroutine profiles from a canary replica and reads allocation hot spots and goroutine counts
- Adds alerts on CPU throttling and on memory against the limit, so the next change shows up before a restart
Outcome
The engagement delivers a table of services with their Go version, runtime settings and limits, a written root-cause analysis for the latency and restart patterns, tuned GOMAXPROCS, GOMEMLIMIT and resource settings, profile-based findings for the owning teams and alerts that watch the container as well as the runtime.