Intermittent 502s that the upstream service logs never saw
Kong
KubernetesNGINX
PostgreSQL
DockerThe intermittent 502s were eliminated, and rolling deployments of upstream services stopped producing an error spike at the gateway.
Overview
A Kong 502 means the gateway could not get a usable response from an upstream target. The hard part is that the upstream is often healthy — the gateway is talking to an address that no longer exists, or has ejected a healthy target because the health check was checking the wrong thing.
Challenge
A small but persistent share of requests returned 502 with no matching entries in the upstream application logs, which pointed away from the application and toward the gateway's view of its targets. The active health check was configured against a path that returned 401 for unauthenticated probes, so Kong's circuit breaker was ejecting healthy targets. Separately, Kong's DNS cache was holding resolved upstream IPs past their TTL, so it kept dialing pods that had been rescheduled.
Environment
Kong gateway on Kubernetes fronting containerized microservices, with upstreams defined by service DNS names and pods rescheduling regularly.
Approach
AceMQ correlated gateway access logs and Kong's own error log against upstream pod lifecycle events to line up 502 bursts with rescheduling. That established the DNS caching behavior as one cause, and the health check endpoint's auth requirement as the other. Both were corrected and validated under a deliberate rolling restart.
Solution
- 1Correlated Kong error logs with upstream pod lifecycle events to time-align 502 bursts with rescheduling
- 2Repointed active health checks at an unauthenticated liveness endpoint returning a true health signal
- 3Tuned DNS resolution and TTL handling so the gateway stops dialing stale upstream addresses
- 4Adjusted passive health check thresholds so transient errors no longer eject healthy targets
- 5Validated the fix under a forced rolling restart of every upstream service
Outcome
The intermittent 502s were eliminated, and rolling deployments of upstream services stopped producing an error spike at the gateway.
Technologies
KongKubernetesNGINXPostgreSQLDockerRelated Use Cases
Kong Rate Limiting Consistency Support
Fixing rate limits that allowed several times the configured quota because the plugin was using the local counter policy across a multi-node gateway.
Kong Gateway Architecture and Migration Consulting
Designing a Kong topology for a company consolidating several ad-hoc API entry points, including control plane separation and environment promotion.
Facing a Kong Production Issue?
AceMQ's senior Kong engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.