A payments platform lost a data node during a hardware fault and the cluster never recovered to green. Replica shards stayed unassigned for days while search traffic ran on single copies, leaving the index one failure away from real data loss.
The surviving nodes had crossed the flood-stage disk watermark, so Elasticsearch had applied a read-only-allow-delete index block and refused to allocate the missing replicas. The allocation explain output pointed at the watermark, but the team could not free space because the block also prevented the delete-by-query cleanup they had planned. Meanwhile the translog on the hot indices kept growing.
Elasticsearch cluster of roughly 20 data nodes on bare metal, hot-warm tiering, several TB of indexed transaction and audit data.
AceMQ engineers started from the allocation explain API rather than guessing, then worked the disk pressure down in an order that let the block clear safely. We moved older indices to the warm tier, force-merged read-only segments to reclaim deleted-document space, then cleared the index block and let allocation resume under a throttled recovery rate so the cluster did not saturate its own network during rebuild.
The cluster returned to green with every replica allocated and no index loss. Because the watermark alerts now fire well before flood stage, the customer has not hit the read-only block again, and node replacement is now a routine maintenance event instead of an incident.
Ongoing 24/7 support for an Elasticsearch estate suffering repeated parent circuit breaker trips and long garbage collection pauses under aggregation load.
Assessment of an oversharded Elasticsearch cluster where cluster-state size and pending task queues were driving master instability.
Whether you need architecture advisory, 24/7 support, or full managed services, AceMQ has the expertise to help.