Overview
This is a typical engagement for a platform team that has opened a Severity 1 case because services inside an Istio mesh return 503 errors, some of them intermittently, and the application teams say nothing changed on their side. An AceMQ engineer joins the incident bridge and works from the proxy logs and the control plane state, with a partner service mesh specialist available for the Istio-specific work.
Challenge
A 503 from a sidecar has several causes that look the same from the application. A DestinationRule that sets no TLS mode defaults to DISABLE, so client sidecars send plain HTTP to servers that expect mutual TLS. Requests to pods of a headless service by pod IP can fail with 503 UC under STRICT mTLS. A proxy that is STALE has not acknowledged the latest configuration. Restarting pods can hide the cause without removing it.
Environment
A Kubernetes cluster with sidecar injection, mesh-wide mTLS in STRICT mode, DestinationRules and VirtualServices owned by several teams, an ingress gateway, and stateful workloads exposed through headless services.
Approach
The engineer asks for four outputs first: istioctl proxy-status, the istio-proxy access log for the failing path with its response flags, the DestinationRule and PeerAuthentication that apply to the failing host, and the namespace labels. The response flag points the next step. NR points to routing, UO to circuit breaking and UF to the upstream connection, and UF leads to the check of TLS modes. Each change is made with the team, with the previous configuration kept as the way back.
Solution
- Access log review with kubectl logs <pod> -c istio-proxy, reading the Envoy response flag that follows the response code
- istioctl proxy-status to see which proxies are SYNCED, NOT SENT or STALE and which istiod revision each one follows
- istioctl proxy-config listeners, routes, clusters and endpoints on a failing pod to compare what the proxy holds with what the team intended
- DestinationRule trafficPolicy.tls.mode checked against the mesh's mTLS mode, with ISTIO_MUTUAL where mutual TLS is on
- Headless service checks: the port name, the Host header and whether clients address pods by their stable DNS names
- Written root cause analysis for the Severity 1 case, with the resource, revision or proxy involved and the configuration change that closed it
Outcome
The engagement delivers a root cause analysis that names the failing resource or proxy, the configuration change that removes the cause, the commands used to confirm it and a short list of preventive changes such as an istioctl analyze check before each merge. The customer keeps the evidence and the runbook for the next incident.