Recovering a domain when the snapshot you relied on will not restore

Overview

An insurance services provider discovered during a DR test that their OpenSearch snapshots would not restore cleanly. Several indices came back red and the restore aborted, which meant their documented recovery plan did not work.

Challenge

The snapshot repository had been reconfigured at some point without a repository verification step, so a portion of the segment files were unreachable from the current bucket path. Restores appeared to start normally and then failed on specific shards. Compounding it, the target domain ran a newer engine version than several of the older snapshots, so some indices were outside the supported restore range and could not be restored directly at all.

Environment

Managed OpenSearch domain on AWS with snapshots stored in object storage, multi-year retention on audit indices.

Approach

We treated the repository as suspect first and verified it before attempting any more restores. Once we knew which snapshots were intact, we reconstructed the recoverable set, restored the too-old indices through an intermediate-version path, and then rebuilt the repository configuration so verification runs as part of the snapshot cycle rather than only during a DR test.

Solution

• Repository verification run against every registered repository to separate intact snapshots from unreachable ones
• Snapshot status inspected shard by shard to identify exactly which indices were recoverable in place
• Older indices restored through a supported intermediate engine version and reindexed forward rather than abandoned
• Repository path and access policy corrected so new snapshots write and read from a single consistent location
• Automated snapshot verification and a scheduled restore test added so failures surface within days, not during a DR exercise
• Recovery runbook rewritten against the procedure that actually worked, with measured timings per index tier

Outcome

Technologies