A telematics provider used a Cassandra table as a work queue, writing rows and deleting them after processing. Read queries began timing out as coordinators scanned tens of thousands of tombstones to return a handful of live rows. AceMQ was engaged under the emergency SLA.
Tombstones remain readable until gc_grace_seconds elapses and compaction removes them, which exists to prevent deleted data from resurrecting on a node that missed the delete. Repairs had not been completing across the ring, so lowering gc_grace_seconds carried real risk of resurrection. The team had been raising the tombstone warning and failure thresholds, which hid the symptom while the underlying scan cost kept growing.
Multi-datacenter Apache Cassandra cluster on bare metal ingesting vehicle telemetry at high volume.
AceMQ separated the immediate relief work from the durable fix. Getting repair completing reliably was the prerequisite for any change to gc_grace_seconds, and the data model change was necessary because a queue on a wide partition generates tombstones faster than compaction can retire them regardless of settings.
Read timeouts on the affected tables stopped, and p99 read latency came down substantially as tombstone scan counts fell by orders of magnitude. Repair now completes on schedule, which also closed the resurrection risk the customer had been carrying unknowingly.
Ongoing support for anti-entropy repair that never completed within gc_grace_seconds, leaving the cluster exposed to deleted data resurrecting.
Redesigning partition keys and clustering order to eliminate unbounded partitions and remove secondary index queries that were hitting every node.
Whether you need architecture advisory, 24/7 support, or full managed services, AceMQ has the expertise to help.