A replication queue that grows after the outage is over

Overview

A telecom operator replicated subscriber session state between two sites with GemFire WAN gateways. A link outage of about 40 minutes left a large queue on the sending site, and when the link returned the queue kept growing instead of draining.

Environment

GemFire 10.x active-active across two data centres, serial gateway senders with persistent queues on shared disks, receivers on the remote site, Java services writing session state.

Approach

AceMQ engineers compared queue size and batch statistics per sender with the receiver's apply rate, found the sender limited by a single dispatcher thread and small batches, and checked that the receiver was not the real bottleneck. Conflation was acceptable for last-value session entries, which cut the volume to move. Changes went in one site at a time with the queue watched throughout.

Solution

1. 01Compared sender queue statistics with receiver apply rate to confirm where the limit was
2. 02Raised dispatcher threads and batch settings in steps, watching receiver latency between each
3. 03Enabled conflation for session regions where only the latest value matters
4. 04Moved the persistent queue disk store to its own storage so queue writes stopped competing with region persistence
5. 05Wrote the site-failover procedure, including what to check about staleness before promoting the remote site
6. 06Added alerts on queue size and queue age, with a threshold set well below the point where failover is risky

Technologies

GemFireJavaObservability