A clear picture of headroom, risk, and what breaks first
Prometheus
KubernetesThe agency received a baseline it had never had, with the items nearest to causing an outage identified and sequenced. Several inherited configuration choices, including consistency levels that did no…
Overview
A public sector agency ran a Cassandra cluster inherited from a vendor engagement that had ended years earlier. Documentation was thin and the operating team had never been given a baseline. AceMQ assessed the cluster against its current workload and projected growth.
Challenge
Assessing an inherited cluster means separating deliberate design choices from accumulated accident. Replication factor and consistency levels needed checking against the durability the agency actually required, token distribution needed verification for hot spots, and JVM and garbage collection behavior needed measurement under real load rather than inference from settings.
Environment
On-premises Apache Cassandra cluster supporting case management and records retention workloads.
Approach
AceMQ collected evidence from the running cluster — token ownership, per-node load distribution, garbage collection pause distributions, compaction backlog, and pending task queues — and evaluated it against the agency's durability and availability requirements. Findings were ranked by how close each was to causing an incident.
Solution
- 1Verified token ownership and per-node load distribution to identify hot spots and unbalanced nodes
- 2Reviewed replication factor and consistency levels against the agency's actual durability requirements
- 3Measured garbage collection pause distribution and heap behavior under production load
- 4Assessed compaction backlog, pending tasks, and SSTable counts per table for accumulating debt
- 5Evaluated repair coverage and completion history against gc_grace_seconds
- 6Projected capacity headroom against growth and delivered a risk-ranked remediation roadmap
Outcome
The agency received a baseline it had never had, with the items nearest to causing an outage identified and sequenced. Several inherited configuration choices, including consistency levels that did not match the stated durability requirement, were corrected as a result.
Technologies
PrometheusKubernetesRelated Use Cases
Cassandra Data Model and Partition Design Consulting
Redesigning partition keys and clustering order to eliminate unbounded partitions and remove secondary index queries that were hitting every node.
Cassandra Repair and Compaction Support
Ongoing support for anti-entropy repair that never completed within gc_grace_seconds, leaving the cluster exposed to deleted data resurrecting.
Ready for a Apache Cassandra Health Check?
AceMQ's senior Apache Cassandra engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.