BlogCassandra

Cassandra Repair, Compaction and Tombstones: Production Operations Guide

Scott Sternloff

By Scott Sternloff, Senior Enterprise Architect

LinkedIn · Updated · 5 min read

On this page

Short answer

Repair, compaction and tombstones are one system in Cassandra: a delete writes a tombstone, repair spreads it to every replica, and compaction removes it after gc_grace_seconds. If any of the three runs late, reads slow down or deleted data comes back. This guide gives the schedule, the commands and the numbers to check. It is written for teams that run their own clusters and want to know when to call for Cassandra support.

Repair, compaction and tombstones are one system in Cassandra: a delete writes a tombstone, repair spreads it to every replica, and compaction removes it after gc_grace_seconds. If any of the three runs late, reads slow down or deleted data comes back. This guide gives the schedule, the commands and the numbers to check. It is written for teams that run their own clusters and want to know when to call for Cassandra support.

How often should you run Cassandra repair?

Run repair on every node at least once inside gc_grace_seconds, which defaults to 864,000 seconds (10 days). The Apache Cassandra docs suggest repairing every node at least every 7 days with that default, which leaves slack for delays. The same page gives a starting point of an incremental repair every 1 to 3 days plus a full repair every 1 to 3 weeks.

CommandWhat it does
nodetool repairIncremental repair (the default type in the 5.0 docs)
nodetool repair --fullFull repair, to catch corruption and operator error
nodetool repair -prOnly the node's primary ranges; run on every node in every datacenter
nodetool repair -prvPreview: estimates streaming without doing it
nodetool repair -vdValidate: compares repaired data across nodes without streaming

Cassandra does not run repair by itself in earlier versions. Cassandra 5.0.8 added CEP-37 Auto Repair, a built-in scheduler, but it needs the JVM property cassandra.autorepair.enable=true set on every node before any schema change, and the project notes say the property cannot be reversed once enabled. Treat it as a planned change, not a toggle.

What to check: a scheduled repair is not a completed one. Compare the last completed repair per table against gc_grace_seconds.

How do you choose a compaction strategy?

Match the strategy to the write pattern, then check that compaction keeps up. The compaction docs list four:

StrategyFitsAvoid when
Unified (UCS)The docs recommend it for most and new workloads, time series or update-heavyYou are on a version without it (it arrived in 5.0)
Size tiered (STCS)Default; general write-heavy tablesRead latency matters and SSTable counts climb
Leveled (LCS)Read-heavy tables, many updates and deletesImmutable time-series data
Time window (TWCS)TTL'd, mostly immutable time seriesTables that are updated or deleted out of order

Three commands show whether compaction is winning:

  • nodetool compactionstats shows active and pending compactions. A pending count that only climbs means compaction is behind.
  • nodetool compactionhistory lists recent compactions, so you can see whether big SSTables are actually being merged.
  • nodetool setcompactionthroughput changes the throttle. The default compaction_throughput is 64MiB/s, and it also applies to validation compactions for repair, so repair and compaction compete for the same budget.

Changing strategy rewrites data: go one table at a time and watch disk headroom.

How do tombstones work, and when do they hurt?

A delete is an insert of a time-stamped marker. A cell with a TTL becomes a tombstone when the TTL ends. Compaction can drop a tombstone only when it is older than gc_grace_seconds and the SSTables holding the older data it shadows are in the same compaction.

The read-path guard rails live in cassandra.yaml: tombstone_warn_threshold is 1000 and tombstone_failure_threshold is 100000 per query. A query that crosses the first logs a warning; one that crosses the second fails. If you see those log lines, find the table before you raise the thresholds.

Checks, in order:

  1. nodetool tablestats <keyspace>.<table> for tombstones scanned per slice and live cells per slice.
  2. tombstone_threshold (default 0.2): the share of an SSTable that must be tombstones before single-SSTable compaction is considered.
  3. unchecked_tombstone_compaction (default false) and tombstone_compaction_interval (default 86,400 seconds) to see why a tombstone-heavy SSTable is not being rewritten.
  4. sstableexpiredblockers when an SSTable holds only tombstones and still will not drop: it lists which other SSTables block it.

Queue-like tables, short TTLs on wide partitions and bulk deletes are the usual sources. Raising thresholds hides the symptom; fix the data model, for example time-bucketed partitions with TTL on a TWCS table.

Why do deleted rows come back?

They come back when a replica misses a delete and the tombstone is purged elsewhere before repair reaches it. The Cassandra docs call this zombie data: if a node stays down longer than gc_grace_seconds, its old copy is treated as new and repaired back into the cluster. Hints are best-effort, and max_hint_window defaults to 3 hours, so a node down for longer needs repair, not hints.

Rules that prevent it:

  • Repair every table inside gc_grace_seconds, including tables nobody reads.
  • Never lower gc_grace_seconds on a table whose repair history you have not verified.
  • After a node has been down longer than the hint window, repair it before it serves traffic you care about.

A weekly operations checklist

CheckCommand or settingHealthy sign
Repair completionrepair history per tableEvery table inside gc_grace_seconds
Compaction backlognodetool compactionstatsPending tasks return to near zero
SSTables per readnodetool tablehistogramsStable, not rising
Tombstone warningssystem logNo new warn or failure lines
Hint backlognodetool tpstats, hints directoryDrains after a node returns

When to call for help

Call when repair has not completed on any table for longer than gc_grace_seconds, when pending compactions have climbed for days, or when read timeouts track a table with heavy deletes. AceMQ provides 24/7 Cassandra support across the Americas, EMEA and APAC, with response targets from 15 minutes, by plan, and support typically runs $24,000-72,000 a year (USD) by SLA, scope and deployment. For the incident view of the same problems, see Cassandra support options, Cassandra pricing and alternatives, and the use cases Cassandra tombstone and read timeout remediation and Cassandra repair and compaction support.

Sources

  • Apache Cassandra docs, Repair: cassandra.apache.org/doc/latest/cassandra/managing/operating/repair.html, read 17 Nov 2026.
  • Apache Cassandra docs, Compaction overview: .../operating/compaction/overview.html, read 17 Nov 2026.
  • Apache Cassandra docs, Tombstones: .../operating/compaction/tombstones.html, read 17 Nov 2026.
  • cassandra.yaml, branch cassandra-5.0 (thresholds, compaction_throughput, max_hint_window): github.com/apache/cassandra/blob/cassandra-5.0/conf/cassandra.yaml, read 17 Nov 2026.
  • Cassandra NEWS.txt, branch cassandra-5.0 (CEP-37 Auto Repair in 5.0.8; full repair after incremental pre-upgrade): github.com/apache/cassandra/blob/cassandra-5.0/NEWS.txt, read 17 Nov 2026.

Frequently Asked Questions

How often should I run nodetool repair?

Every node must finish a repair inside `gc_grace_seconds` (10 days by default). The Apache docs suggest every 7 days at least, as incremental every 1 to 3 days with a full repair every 1 to 3 weeks.

What is the default compaction strategy in Cassandra?

Size tiered (STCS) is the default. For new workloads the 5.0 docs recommend the Unified Compaction Strategy; TWCS is for TTL'd time series.

What are the default tombstone thresholds?

`tombstone_warn_threshold` is 1000 and `tombstone_failure_threshold` is 100000 in `cassandra.yaml` for 5.0.

Is it safe to lower gc_grace_seconds?

Only if every repair on that table completes well inside the new value. Otherwise you risk resurrected deletes.

Does Cassandra 5.0 run repair automatically?

From 5.0.8, an optional Auto Repair scheduler exists (CEP-37). It needs a JVM property set on all nodes and cannot be switched back off at that level.

Scott Sternloff

Author

Scott SternloffSenior Enterprise Architect

Senior Enterprise Architect at AceMQ. Writes the technical coverage here: RabbitMQ, Kafka and Redis architecture, clustering and migrations.

Talk to an engineer

Need a hand with Cassandra?

Senior engineers who run it in production. Get a quote or a free consultation, no sales pitch.

Free Consultation

Get Expert Eyes on Your Cassandra Environment

Whether you're troubleshooting a production incident, planning a migration, or want a second opinion on your architecture — our team is ready. No pitch, just answers.

Email Us