Intermittent 502s that the upstream service logs never saw

Overview

A Kong 502 means the gateway could not get a usable response from an upstream target. The hard part is that the upstream is often healthy — the gateway is talking to an address that no longer exists, or has ejected a healthy target because the health check was checking the wrong thing.

Challenge

A small but persistent share of requests returned 502 with no matching entries in the upstream application logs, which pointed away from the application and toward the gateway's view of its targets. The active health check was configured against a path that returned 401 for unauthenticated probes, so Kong's circuit breaker was ejecting healthy targets. Separately, Kong's DNS cache was holding resolved upstream IPs past their TTL, so it kept dialing pods that had been rescheduled.

Environment

Kong gateway on Kubernetes fronting containerized microservices, with upstreams defined by service DNS names and pods rescheduling regularly.

Approach

AceMQ correlated gateway access logs and Kong's own error log against upstream pod lifecycle events to line up 502 bursts with rescheduling. That established the DNS caching behavior as one cause, and the health check endpoint's auth requirement as the other. Both were corrected and validated under a deliberate rolling restart.

Solution

• Correlated Kong error logs with upstream pod lifecycle events to time-align 502 bursts with rescheduling
• Repointed active health checks at an unauthenticated liveness endpoint returning a true health signal
• Tuned DNS resolution and TTL handling so the gateway stops dialing stale upstream addresses
• Adjusted passive health check thresholds so transient errors no longer eject healthy targets
• Validated the fix under a forced rolling restart of every upstream service

Outcome

Technologies