Repair that finishes, on a schedule you can prove

Overview

A payments processor ran repair as a cron job that invoked a full repair across all keyspaces. Runs regularly exceeded the gc_grace_seconds window, overlapped with each other, and were killed when they saturated cluster I/O. Nobody could state when the ring had last been fully repaired. AceMQ took over support with named senior engineers.

Challenge

Repair that does not complete inside gc_grace_seconds is not a housekeeping problem — it means deleted data can come back after compaction retires tombstones on some replicas but not others. Full repairs across a large multi-datacenter ring were too expensive to complete, and the streaming they triggered competed directly with production traffic.

Environment

Multi-datacenter Apache Cassandra spanning on-premises and cloud, backing transaction ledger and settlement data.

Approach

AceMQ replaced whole-ring repair with scheduled, segmented, incremental repair that could be paced against available cluster headroom and resumed after interruption. Compaction strategy and throughput settings were retuned per table so repair and compaction stopped competing for the same I/O.

Solution

• Replaced cron-driven full repairs with a scheduled, segmented repair orchestrator that resumes rather than restarts
• Paced repair against available cluster headroom so it no longer saturated I/O during business hours
• Retuned compaction strategy per table against actual read, write, and delete patterns rather than the default
• Adjusted compaction throughput and concurrent compactor settings to match the underlying storage
• Instrumented repair completion time per keyspace against gc_grace_seconds as a tracked, alertable metric
• Provided 24/7 escalation to senior Cassandra engineers with no tier-1 triage in the path

Outcome

Technologies