Recovering a domain when the snapshot you relied on will not restore
All audit-critical indices were recovered and the domain returned to green. The customer now has a restore path that has been exercised end to end, and snapshot verification failures page the on-call …
Overview
An insurance services provider discovered during a DR test that their OpenSearch snapshots would not restore cleanly. Several indices came back red and the restore aborted, which meant their documented recovery plan did not work.
Challenge
The snapshot repository had been reconfigured at some point without a repository verification step, so a portion of the segment files were unreachable from the current bucket path. Restores appeared to start normally and then failed on specific shards. Compounding it, the target domain ran a newer engine version than several of the older snapshots, so some indices were outside the supported restore range and could not be restored directly at all.
Environment
Managed OpenSearch domain on AWS with snapshots stored in object storage, multi-year retention on audit indices.
Approach
We treated the repository as suspect first and verified it before attempting any more restores. Once we knew which snapshots were intact, we reconstructed the recoverable set, restored the too-old indices through an intermediate-version path, and then rebuilt the repository configuration so verification runs as part of the snapshot cycle rather than only during a DR test.
Solution
- 1Repository verification run against every registered repository to separate intact snapshots from unreachable ones
- 2Snapshot status inspected shard by shard to identify exactly which indices were recoverable in place
- 3Older indices restored through a supported intermediate engine version and reindexed forward rather than abandoned
- 4Repository path and access policy corrected so new snapshots write and read from a single consistent location
- 5Automated snapshot verification and a scheduled restore test added so failures surface within days, not during a DR exercise
- 6Recovery runbook rewritten against the procedure that actually worked, with measured timings per index tier
Outcome
All audit-critical indices were recovered and the domain returned to green. The customer now has a restore path that has been exercised end to end, and snapshot verification failures page the on-call rather than sitting silent until the next DR test.
Technologies
Related Use Cases
OpenSearch Cluster Support for Regulated Workloads
24/7 enterprise support for OpenSearch clusters carrying regulated search and audit workloads, including security plugin and upgrade coverage.
Elasticsearch to OpenSearch Migration Assessment
Assessment of the technical and licensing implications of moving a large Elasticsearch estate to OpenSearch, including client and plugin compatibility.
Facing a OpenSearch Production Issue?
AceMQ's senior OpenSearch engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.