The slow call was fanning out to every member because the data lived apart

Overview

A card processor used GemFire to hold accounts, authorisation holds and transaction history, and ran a balance function against them on every authorisation. As volumes grew, the function began to time out at peak, and adding members made it no better.

Environment

GemFire 10.x on virtual machines: 8 servers, 3 locators, partitioned regions for accounts, holds and history, a small replicated region for reference data, Spring Boot services calling functions through the client API.

Approach

AceMQ engineers mapped each function and query to the keys it touched, chose the account identifier as the partitioning key with a partition resolver, and designed new colocated regions plus replicated regions for the small reference sets. Data was backfilled by function into the new regions while services dual wrote, then reads were switched and the old regions removed.

Solution

1. 01Mapped every function and OQL query to the keys and regions it reads
2. 02Chose the account identifier as the partitioning key and implemented a partition resolver for related regions
3. 03Created new regions colocated with accounts, and replicated regions for small reference data
4. 04Backfilled history into the new regions with a function while services dual wrote
5. 05Rewrote the balance function to execute on a single member with a key filter
6. 06Compared results side by side before switching reads, then dropped the old regions

Technologies

GemFireJavaSpring