Back to all use cases

RemediationFinancial Services / Trading

The member did not crash; its JVM paused long enough to be voted out

Regional Electronic Trading Platform

GemFire members kept leaving the distributed system around market open with no crash and no out-of-memory error. AceMQ traced it to garbage collection pauses longer than the failure-detection window and retuned heap and thresholds together.

Overview

A trading platform held reference data and positions in GemFire partitioned regions on bare-metal servers. Several times a week, around market open, a member was declared dead and the cluster spent a long time rebalancing while order latency climbed.

Challenge

Logs said the member had unexpectedly left the distributed system, but the process was still running and memory looked fine. Raising member-timeout had hidden the symptom for a while and then made real failures slower to detect. Each departure triggered redundancy recovery and rebalancing at exactly the time the cluster was busiest.

Environment

GemFire 9.15 on bare-metal Linux servers: 6 data members, 3 locators on separate hosts, 64 GB heaps with the G1 collector, partitioned regions with one redundant copy, a Java order service as the main client.

Approach

AceMQ engineers lined up GC logs, locator suspect-processing entries and host network counters on a single timeline and found that every departure matched a full GC pause longer than the failure-detection window. They then worked on the heap rather than the timeout: smaller heaps, off-heap for the largest regions, and resource manager thresholds that shed load before a collection became pathological.

Solution

  1. Correlated GC logs, locator logs and network counters on one timeline to separate pauses from network faults
  2. Reduced heap per member and moved the largest regions off-heap so object count stayed manageable for the collector
  3. Set eviction and critical heap thresholds, with overflow to disk for regions that must not lose entries
  4. Returned member-timeout to a deliberate value instead of the inflated one
  5. Staggered rebalancing so a departure no longer triggered a full rebalance in the middle of the open
  6. Added alerts on pause length and heap after collection, so drift shows up before a member leaves

Outcome

In the two months after the change the cluster recorded no unexpected member departures during trading hours. Pause length became a graph the team watched, and member-timeout stopped being a number someone raised whenever the cluster complained.

Technologies

GemFireJavaLinux
  • Assessment

    GemFire Cluster and Region Architecture Assessment

    Enterprise Technology OrganizationsCross-Industry

    A structured review of GemFire region design, WAN topology, JVM tuning and security configuration, producing a risk-prioritized roadmap before problems reach production.

    GemFireJavaSpring
    Read case study: GemFire Cluster and Region Architecture Assessment
  • Support

    Multi-Technology Support Including Redis for Global Trading Firm

    DRW (Global Trading Firm)Financial Services

    AceMQ expanded its enterprise support agreement with global quantitative trading firm DRW to include Redis caching alongside RabbitMQ, providing L3 escalation support across the full messaging and caching technology stack.

    RedisRabbitMQ
    Read case study: Multi-Technology Support Including Redis for Global Trading Firm
  • Support

    Redis Memory and Eviction Policy Support

    Healthcare Analytics ProviderHealthcare

    Ongoing support for instances where the eviction policy did not match how the keyspace was used, causing session data to be evicted under memory pressure.

    RedisKubernetesPrometheusTerraform
    Read case study: Redis Memory and Eviction Policy Support
  • Support

    GemFire WAN Gateway Queue That Never Caught Up

    Regional Telecom OperatorTelecommunications

    After a link outage, a GemFire gateway sender queue between two sites kept growing instead of draining. AceMQ tuned the sender and receiver, drained the backlog and set up support for the replication path.

    GemFireJavaObservability
    Read case study: GemFire WAN Gateway Queue That Never Caught Up
  • Consulting

    GemFire Region Colocation Redesign for Slow Function Calls

    European Card ProcessorFinancial Services

    Function executions and queries across non-colocated partitioned regions were timing out under load. AceMQ redesigned the region layout around the access pattern and migrated the data with a side-by-side run.

    GemFireJavaSpring
    Read case study: GemFire Region Colocation Redesign for Slow Function Calls

Related Reading

Facing a GemFire Production Issue?

AceMQ's senior GemFire engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.