Fixing the access pattern, not just raising the tombstone threshold
Kubernetes
PrometheusLinuxRead timeouts on the affected tables stopped, and p99 read latency came down substantially as tombstone scan counts fell by orders of magnitude. Repair now completes on schedule, which also closed the…
Overview
A telematics provider used a Cassandra table as a work queue, writing rows and deleting them after processing. Read queries began timing out as coordinators scanned tens of thousands of tombstones to return a handful of live rows. AceMQ was engaged under the emergency SLA.
Challenge
Tombstones remain readable until gc_grace_seconds elapses and compaction removes them, which exists to prevent deleted data from resurrecting on a node that missed the delete. Repairs had not been completing across the ring, so lowering gc_grace_seconds carried real risk of resurrection. The team had been raising the tombstone warning and failure thresholds, which hid the symptom while the underlying scan cost kept growing.
Environment
Multi-datacenter Apache Cassandra cluster on bare metal ingesting vehicle telemetry at high volume.
Approach
AceMQ separated the immediate relief work from the durable fix. Getting repair completing reliably was the prerequisite for any change to gc_grace_seconds, and the data model change was necessary because a queue on a wide partition generates tombstones faster than compaction can retire them regardless of settings.
Solution
- 1Identified the specific partitions and query shapes scanning the largest tombstone counts
- 2Restored reliable repair completion across the ring as a prerequisite for touching gc_grace_seconds
- 3Moved the queue workload off the wide-partition pattern to time-bucketed partitions with TTL-based expiry
- 4Switched the affected tables to a compaction strategy matched to the write and delete pattern
- 5Reverted the raised tombstone thresholds so the condition surfaces again rather than being suppressed
- 6Ran targeted compaction to retire the existing tombstone backlog during a controlled window
Outcome
Read timeouts on the affected tables stopped, and p99 read latency came down substantially as tombstone scan counts fell by orders of magnitude. Repair now completes on schedule, which also closed the resurrection risk the customer had been carrying unknowingly.
Technologies
KubernetesPrometheusLinuxRelated Use Cases
Cassandra Repair and Compaction Support
Ongoing support for anti-entropy repair that never completed within gc_grace_seconds, leaving the cluster exposed to deleted data resurrecting.
Cassandra Data Model and Partition Design Consulting
Redesigning partition keys and clustering order to eliminate unbounded partitions and remove secondary index queries that were hitting every node.
Facing a Apache Cassandra Production Issue?
AceMQ's senior Apache Cassandra engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.