Retention that satisfies auditors without paying hot-tier prices for cold data

Overview

A healthcare analytics provider needed multi-year retention for audit reasons but was keeping everything on expensive hot nodes. AceMQ designed an ILM and tiering strategy that meets the retention requirement while moving the bulk of the data onto cheaper storage.

Challenge

Retention rules were enforced by hand-written cleanup scripts that occasionally deleted the wrong index pattern. There was no warm or cold tier, snapshots were taken inconsistently, and nobody had tested a restore. The compliance team could not answer how long a given record class was actually retained, because the answer depended on which script had last run successfully.

Environment

Hybrid Elasticsearch deployment with on-premises hot nodes and cloud object storage available for snapshots and searchable data.

Approach

We started by mapping each index pattern to its actual regulatory retention obligation, since several patterns were being kept far longer than required. From there we designed rollover-driven ILM policies with explicit hot, warm, and cold phases, added a snapshot lifecycle policy against object storage, and rehearsed restore procedures until timings were known rather than assumed.

Solution

• Retention requirements mapped per index pattern and reconciled against what the cleanup scripts were actually doing
• ILM policies built on rollover aliases with size and age conditions, replacing hand-rolled deletion scripts entirely
• Hot, warm, and cold phase definitions including force-merge, replica reduction, and shard allocation filtering by node attribute
• Snapshot lifecycle management to object storage with retention rules that match the compliance obligation
• Restore drills executed and timed so the recovery objective is a measured number rather than an estimate
• Policy definitions committed as infrastructure code so changes go through review

Outcome

Technologies