Throttle limits that hold on one node and not the next

Overview

WSO2 API Manager can evaluate throttling either locally per gateway node or centrally through the traffic manager. When gateway nodes lose their connection to the traffic manager, or when the event topics between them are misconfigured, throttling decisions diverge and the enforced limit stops matching the published policy.

Challenge

Consumers on a metered tier were exceeding their subscription limits, but only intermittently and not reproducibly on any single node. Gateway nodes had been scaled up, and the new nodes' traffic manager event receiver configuration did not match the originals, so throttle decision events were not reaching all of them. Under load, the divergence widened.

Environment

WSO2 API Manager on Kubernetes with multiple gateway nodes, a traffic manager tier, and a shared user store.

Approach

AceMQ traced throttle decision events from the traffic manager through to each gateway node and identified the nodes that were not receiving them. We corrected the event receiver configuration, then validated throttle consistency under load by driving traffic deliberately at each node.

Solution

• Traced throttle decision event flow from traffic manager to each gateway node to find the gaps
• Corrected event receiver and topic configuration on gateway nodes added during scale-out
• Aligned throttle policy tiers and key templates so policies evaluate identically across nodes
• Added failure behavior for gateway nodes that lose traffic manager connectivity
• Validated enforced limits under load with traffic driven at each node individually
• Templated gateway node configuration so future scale-out cannot drift

Outcome

Technologies