Repair that finishes, on a schedule you can prove
Prometheus
KubernetesRepair now completes well inside gc_grace_seconds on every keyspace, with completion tracked as a metric rather than assumed. The resurrection risk the cluster had been carrying is closed, and repair …
Overview
A payments processor ran repair as a cron job that invoked a full repair across all keyspaces. Runs regularly exceeded the gc_grace_seconds window, overlapped with each other, and were killed when they saturated cluster I/O. Nobody could state when the ring had last been fully repaired. AceMQ took over support with named senior engineers.
Challenge
Repair that does not complete inside gc_grace_seconds is not a housekeeping problem — it means deleted data can come back after compaction retires tombstones on some replicas but not others. Full repairs across a large multi-datacenter ring were too expensive to complete, and the streaming they triggered competed directly with production traffic.
Environment
Multi-datacenter Apache Cassandra spanning on-premises and cloud, backing transaction ledger and settlement data.
Approach
AceMQ replaced whole-ring repair with scheduled, segmented, incremental repair that could be paced against available cluster headroom and resumed after interruption. Compaction strategy and throughput settings were retuned per table so repair and compaction stopped competing for the same I/O.
Solution
- 1Replaced cron-driven full repairs with a scheduled, segmented repair orchestrator that resumes rather than restarts
- 2Paced repair against available cluster headroom so it no longer saturated I/O during business hours
- 3Retuned compaction strategy per table against actual read, write, and delete patterns rather than the default
- 4Adjusted compaction throughput and concurrent compactor settings to match the underlying storage
- 5Instrumented repair completion time per keyspace against gc_grace_seconds as a tracked, alertable metric
- 6Provided 24/7 escalation to senior Cassandra engineers with no tier-1 triage in the path
Outcome
Repair now completes well inside gc_grace_seconds on every keyspace, with completion tracked as a metric rather than assumed. The resurrection risk the cluster had been carrying is closed, and repair no longer degrades production latency while it runs.
Technologies
PrometheusKubernetesRelated Use Cases
Cassandra Tombstone Accumulation and Read Timeout Remediation
Resolving read timeouts caused by tombstone accumulation on queue-like partitions where deletes outpaced compaction and gc_grace_seconds.
Cassandra Cluster Health and Capacity Assessment
Assessment covering topology, replication and consistency configuration, JVM and garbage collection behavior, compaction health, and growth headroom.
Need Expert Apache Cassandra Support?
AceMQ's senior Apache Cassandra engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.