Retention that satisfies auditors without paying hot-tier prices for cold data
ElasticsearchKibanaAmazon S3TerraformThe majority of indexed data now lives on warm and cold tiers, freeing substantial hot-node capacity for query workloads. Retention is enforced by policy instead of scripts, and the compliance team ca…
Overview
A healthcare analytics provider needed multi-year retention for audit reasons but was keeping everything on expensive hot nodes. AceMQ designed an ILM and tiering strategy that meets the retention requirement while moving the bulk of the data onto cheaper storage.
Challenge
Retention rules were enforced by hand-written cleanup scripts that occasionally deleted the wrong index pattern. There was no warm or cold tier, snapshots were taken inconsistently, and nobody had tested a restore. The compliance team could not answer how long a given record class was actually retained, because the answer depended on which script had last run successfully.
Environment
Hybrid Elasticsearch deployment with on-premises hot nodes and cloud object storage available for snapshots and searchable data.
Approach
We started by mapping each index pattern to its actual regulatory retention obligation, since several patterns were being kept far longer than required. From there we designed rollover-driven ILM policies with explicit hot, warm, and cold phases, added a snapshot lifecycle policy against object storage, and rehearsed restore procedures until timings were known rather than assumed.
Solution
- 1Retention requirements mapped per index pattern and reconciled against what the cleanup scripts were actually doing
- 2ILM policies built on rollover aliases with size and age conditions, replacing hand-rolled deletion scripts entirely
- 3Hot, warm, and cold phase definitions including force-merge, replica reduction, and shard allocation filtering by node attribute
- 4Snapshot lifecycle management to object storage with retention rules that match the compliance obligation
- 5Restore drills executed and timed so the recovery objective is a measured number rather than an estimate
- 6Policy definitions committed as infrastructure code so changes go through review
Outcome
The majority of indexed data now lives on warm and cold tiers, freeing substantial hot-node capacity for query workloads. Retention is enforced by policy instead of scripts, and the compliance team can answer retention questions from the policy definitions directly.
Technologies
ElasticsearchKibanaAmazon S3TerraformRelated Use Cases
Elasticsearch Shard and Cluster State Assessment
Assessment of an oversharded Elasticsearch cluster where cluster-state size and pending task queues were driving master instability.
Elasticsearch Heap Pressure and Circuit Breaker Support
Ongoing 24/7 support for an Elasticsearch estate suffering repeated parent circuit breaker trips and long garbage collection pauses under aggregation load.
Need Elasticsearch Architecture Guidance?
AceMQ's senior Elasticsearch engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.