Finding the plugin that costs more than the service behind it

Overview

Every Kong plugin runs in the request path. A chain assembled over two years by different teams can easily add more latency than the upstream service it fronts, and custom Lua plugins making blocking network calls are the usual culprits.

Challenge

Gateway p99 latency had grown steadily without any single deployment to blame. The plugin chain included several auth-related plugins with overlapping responsibilities, a custom Lua plugin performing a synchronous lookup against an external service, and logging plugins writing to a sink that intermittently backed up.

Environment

Kong on Kubernetes in front of payment authorization services with strict end-to-end latency budgets.

Approach

AceMQ instrumented the gateway to attribute latency per plugin per phase rather than treating the chain as a single number, then ran controlled comparisons with individual plugins disabled. That separated the plugins that cost real time from the ones that merely looked suspicious.

Solution

• Attributed request latency per plugin and per execution phase rather than to the chain as a whole
• Ran controlled A/B comparisons with individual plugins disabled under production-like load
• Identified a custom Lua plugin making a blocking external call in the access phase
• Consolidated overlapping authentication plugins into a single verified path
• Recommended asynchronous or buffered logging where a backed-up sink was adding tail latency
• Delivered a latency budget per plugin that new plugins must fit within

Outcome

Technologies