Getting a stalled Elasticsearch cluster back to green without data loss
ElasticsearchKibanaLinux
KubernetesThe cluster returned to green with every replica allocated and no index loss. Because the watermark alerts now fire well before flood stage, the customer has not hit the read-only block again, and nod…
Overview
A payments platform lost a data node during a hardware fault and the cluster never recovered to green. Replica shards stayed unassigned for days while search traffic ran on single copies, leaving the index one failure away from real data loss.
Challenge
The surviving nodes had crossed the flood-stage disk watermark, so Elasticsearch had applied a read-only-allow-delete index block and refused to allocate the missing replicas. The allocation explain output pointed at the watermark, but the team could not free space because the block also prevented the delete-by-query cleanup they had planned. Meanwhile the translog on the hot indices kept growing.
Environment
Elasticsearch cluster of roughly 20 data nodes on bare metal, hot-warm tiering, several TB of indexed transaction and audit data.
Approach
AceMQ engineers started from the allocation explain API rather than guessing, then worked the disk pressure down in an order that let the block clear safely. We moved older indices to the warm tier, force-merged read-only segments to reclaim deleted-document space, then cleared the index block and let allocation resume under a throttled recovery rate so the cluster did not saturate its own network during rebuild.
Solution
- 1Read cluster allocation explain output for each unassigned shard to confirm the watermark, not corruption, was the blocker
- 2Relocated aged indices to warm-tier nodes and force-merged read-only indices to reclaim space from deleted documents
- 3Cleared the read-only-allow-delete block only after headroom was restored, avoiding a repeat trip over flood stage
- 4Throttled indices.recovery.max_bytes_per_sec during rebuild so replica recovery did not starve live query traffic
- 5Set explicit low/high/flood watermarks with realistic headroom and added disk alerting ahead of the high watermark
- 6Documented the recovery runbook so the customer's on-call can clear the same condition without escalation
Outcome
The cluster returned to green with every replica allocated and no index loss. Because the watermark alerts now fire well before flood stage, the customer has not hit the read-only block again, and node replacement is now a routine maintenance event instead of an incident.
Technologies
ElasticsearchKibanaLinuxKubernetesRelated Use Cases
Elasticsearch Heap Pressure and Circuit Breaker Support
Ongoing 24/7 support for an Elasticsearch estate suffering repeated parent circuit breaker trips and long garbage collection pauses under aggregation load.
Elasticsearch Shard and Cluster State Assessment
Assessment of an oversharded Elasticsearch cluster where cluster-state size and pending task queues were driving master instability.
Facing a Elasticsearch Production Issue?
AceMQ's senior Elasticsearch engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.