Back to all use cases

RemediationSoftware / Digital Platforms

Which Go version a service was built with decides how many threads it thinks it may use

Platform Engineering Team

A typical engagement for Go services that show latency spikes under a CPU limit or restart with exit code 137: compare the Go version and GOMAXPROCS with the container limits, read heap and goroutine profiles, and set GOMEMLIMIT and requests to match.

Golang consulting

Overview

This is a typical engagement for a platform team that runs a few dozen Go services on Kubernetes with CPU limits set. Some show p99 latency spikes that do not follow traffic, and a few restart with exit code 137 under load. The services were built over several years with different Go versions. AceMQ and a partner Go specialist find where the time and memory go, using the service's own profiles.

Challenge

Go 1.25 changed the default GOMAXPROCS on Linux to respect the container's CPU limit, and Kubernetes CPU requests are ignored by that logic. A service built with an older Go, or with GOMAXPROCS set by hand, can run more threads than its quota allows and spend its time throttled. Exit code 137 means the kernel killed the process, so no Go panic appears, and heap graphs alone do not explain it.

Environment

Go services on a managed Kubernetes cluster, built with Go versions from several releases, CPU and memory limits set per deployment, Prometheus metrics for container CPU throttling and memory, and net/http/pprof endpoints on some services.

Approach

AceMQ lists the Go version each binary was built with (go version -m reads it from the binary), the GOMAXPROCS and GOMEMLIMIT settings in each deployment and the container limits. It then compares CPU throttling and working-set memory over time with Go's own runtime metrics, and takes CPU, heap and goroutine profiles from a canary replica rather than the whole fleet.

Solution

  1. Reads the Go version from each binary and the GOMAXPROCS and GOMEMLIMIT settings from each deployment
  2. Plots container CPU throttling next to request latency to show which services are limited by their quota
  3. Removes hand-set GOMAXPROCS values where the Go version already follows the cgroup limit, and sets it deliberately where it does not
  4. Sets GOMEMLIMIT below the container memory limit with headroom for memory outside the Go heap
  5. Takes CPU, heap and goroutine profiles from a canary replica and reads allocation hot spots and goroutine counts
  6. Adds alerts on CPU throttling and on memory against the limit, so the next change shows up before a restart

Outcome

The engagement delivers a table of services with their Go version, runtime settings and limits, a written root-cause analysis for the latency and restart patterns, tuned GOMAXPROCS, GOMEMLIMIT and resource settings, profile-based findings for the owning teams and alerts that watch the container as well as the runtime.

Technologies

Go (Golang)KubernetesPrometheusDocker

Facing a Go (Golang) Production Issue?

AceMQ's senior Go (Golang) engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.