Finding the plugin that costs more than the service behind it
Gateway-added latency at p99 dropped substantially once the blocking custom plugin was reworked and the auth chain was consolidated, and the team gained a latency budget to evaluate future plugins aga…
Overview
Every Kong plugin runs in the request path. A chain assembled over two years by different teams can easily add more latency than the upstream service it fronts, and custom Lua plugins making blocking network calls are the usual culprits.
Challenge
Gateway p99 latency had grown steadily without any single deployment to blame. The plugin chain included several auth-related plugins with overlapping responsibilities, a custom Lua plugin performing a synchronous lookup against an external service, and logging plugins writing to a sink that intermittently backed up.
Environment
Kong on Kubernetes in front of payment authorization services with strict end-to-end latency budgets.
Approach
AceMQ instrumented the gateway to attribute latency per plugin per phase rather than treating the chain as a single number, then ran controlled comparisons with individual plugins disabled. That separated the plugins that cost real time from the ones that merely looked suspicious.
Solution
- 1Attributed request latency per plugin and per execution phase rather than to the chain as a whole
- 2Ran controlled A/B comparisons with individual plugins disabled under production-like load
- 3Identified a custom Lua plugin making a blocking external call in the access phase
- 4Consolidated overlapping authentication plugins into a single verified path
- 5Recommended asynchronous or buffered logging where a backed-up sink was adding tail latency
- 6Delivered a latency budget per plugin that new plugins must fit within
Outcome
Gateway-added latency at p99 dropped substantially once the blocking custom plugin was reworked and the auth chain was consolidated, and the team gained a latency budget to evaluate future plugins against.
Technologies
Related Use Cases
Kong Gateway Architecture and Migration Consulting
Designing a Kong topology for a company consolidating several ad-hoc API entry points, including control plane separation and environment promotion.
Kong 502 and Upstream Health Check Remediation
Tracing intermittent 502s at the Kong gateway to misconfigured active health checks and stale DNS resolution of upstream service names.
Ready for a Kong Health Check?
AceMQ's senior Kong engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.