Overview
A platform team can run a Cassandra ring well until repair windows, node replacements and upgrades compete with product work for the same two people. This scenario describes how AceMQ's Tier 3 managed Cassandra runs on infrastructure the customer owns, with system access under the customer's change process.
Challenge
The team runs subscriber-facing data on a multi-datacenter Apache Cassandra ring. Repair is scheduled but nobody can show that every table finishes inside gc_grace_seconds, compaction backlogs appear after bulk loads, and a node replacement has never been rehearsed.
Environment
Apache Cassandra 4.1 on virtual machines in the customer's own data centres, two datacenters with NetworkTopologyStrategy, LOCAL_QUORUM reads and writes, monitoring in place but alerting only to a shared mailbox, and a change process the customer's security team controls.
Approach
Onboarding starts with an assessment of version and patch position, repair history per table, partition size distribution, tombstone counts, compaction settings and backup evidence. Active risks are fixed first. AceMQ then takes the pager under the contracted SLA, working through the customer's access controls and change process, with every change logged.
Solution
- Assesses repair completion per table against gc_grace_seconds, partition sizes and tombstone counts before taking over
- Moves repair to a scheduler running subrange repairs, with an alert on any table whose run does not finish
- Writes and rehearses runbooks for node replacement, datacenter expansion and a rolling upgrade, on a clone first
- Sets up monitoring for pending compactions, dropped messages, hint backlog, GC pauses and per-table latency, routed to AceMQ's on-call
- Schedules restore tests into a scratch ring and records the results; keeps runbooks and topology documentation as the customer's property
Outcome
The engagement delivers an assessed ring, active risks remediated, repair that is checked for completion, rehearsed replacement and upgrade runbooks, monitoring and on-call run by AceMQ, and a handover pack the customer can use if operations come back in-house. Hosts, data and keys stay with the customer throughout.
Technologies
Apache Cassandra
Kubernetes
Prometheus
Linux
Kafka
Docker
Kong
PostgreSQL
