Back to all use cases

SupportFinancial Services / Payments

A reshard that stops halfway is worse than one that never started

Regional Payments Processor

A Valkey Cluster resharding ahead of a seasonal peak stalled with slots left in migrating and importing states. AceMQ resolved the slots on both nodes, finished the move in controlled batches and set up support for the next one.

Overview

A payments processor ran Valkey 8.0 in cluster mode for idempotency keys and session state. Before a seasonal volume peak the team added shards and started a reshard, which stopped partway with traffic still flowing.

Challenge

Several slots were left in migrating and importing states, clients saw ASK and MOVED redirects for a subset of keys, and tail latency rose on two shards. Retrying the reshard script made the state worse, because the source and target nodes disagreed about who owned the slots. The cause sat in a few very large hash keys that the key-by-key migration could not move inside the configured timeout.

Environment

Valkey 8.0 Cluster on Kubernetes in AWS: 6 primaries with one replica each, going to 9, StatefulSets with persistent volumes, Prometheus for metrics, clients on Java and Go.

Approach

AceMQ engineers stopped the retries, read the slot state on both source and target nodes with CLUSTER NODES and CLUSTER SETSLOT, and found the keys blocking each stuck slot. The large hashes were moved or split in a controlled window, the open slots were closed explicitly on both ends, and the remaining reshard ran in small batches with latency watched between them.

Solution

  1. Paused the reshard automation and recorded the slot state on every node before changing anything
  2. Identified the large hash keys blocking migration with a keyspace sample and MEMORY USAGE
  3. Closed each migrating and importing slot explicitly on both source and target nodes
  4. Split the largest hashes into bounded buckets in the application and migrated the rest in batches
  5. Added a pre-check for large keys and a per-batch latency gate to the reshard runbook
  6. Planned the move to Valkey 9.0, where atomic slot migration moves a whole slot at once

Outcome

The cluster reached nine shards with all 16,384 slots owned and no open migrations before the peak. The team kept a runbook for large-key checks and batched resharding, and an upgrade to atomic slot migration on Valkey 9.0 went into the next maintenance plan.

Technologies

ValkeyKubernetesPrometheus
  • Remediation

    Redis to Valkey Migration Under Memory Pressure

    Digital Collaboration PlatformSoftware

    A production Redis cluster was crashing under memory growth traced to its full-text index rather than to the workload. AceMQ diagnosed the leak, stabilized the cluster, and mapped the Valkey migration path.

    ValkeyRedisKubernetes
    Read case study: Redis to Valkey Migration Under Memory Pressure
  • Assessment

    Redis Cluster Health and Architecture Assessment

    Enterprise Technology OrganizationsCross-Industry

    AceMQ's Redis Health and Architecture Assessment identifies cluster vulnerabilities, performance bottlenecks, and optimization opportunities before they become production incidents — delivering a prioritized remediation roadmap.

    Redis
    Read case study: Redis Cluster Health and Architecture Assessment
  • Consulting

    Redis Cluster Mode Migration Consulting

    Global Payments ProcessorFinancial Services

    Planning a migration from vertically scaled standalone Redis to Cluster mode, including hash tag design and remediation of cross-slot operations.

    RedisKubernetesTerraformPrometheus
    Read case study: Redis Cluster Mode Migration Consulting
  • Support

    Prometheus Restart and WAL Replay Support

    Global Payments ProcessorFinancial Services

    Ongoing support for large Prometheus instances where restarts caused extended monitoring blind spots due to slow write-ahead log replay.

    PrometheusThanosKubernetesGrafana
    Read case study: Prometheus Restart and WAL Replay Support
  • Support

    Cassandra Repair and Compaction Support

    Global Payments ProcessorFinancial Services

    Ongoing support for anti-entropy repair that never completed within gc_grace_seconds, leaving the cluster exposed to deleted data resurrecting.

    Apache CassandraReaperPrometheusKubernetes
    Read case study: Cassandra Repair and Compaction Support
  • Remediation

    Tanzu Postgres Failover That Never Triggered

    Regional Payments ProcessorFinancial Services

    Finding why a Tanzu Postgres primary stayed in service while refusing queries, and fixing the health probe so the operator promotes a replica.

    Tanzu PostgresPostgreSQLKubernetesPrometheus
    Read case study: Tanzu Postgres Failover That Never Triggered

Related Reading

Need Expert Valkey Support?

AceMQ's senior Valkey engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.