Finding out which fields you are indexing that nobody ever searches
ElasticsearchKibanaLogstashThe assessment showed that a large share of indexed fields had never appeared in any query. With explicit mappings and tiered retention, projected storage growth flattened enough to cancel the next pl…
Overview
An insurance services provider's ELK cluster had outgrown its storage twice in eighteen months with no clear reason. AceMQ assessed what was actually being indexed and how much of it anyone used.
Challenge
Default dynamic mapping meant every field in every JSON log line was both indexed and stored, including large nested structures nobody had ever searched. Several services logged full request and response bodies at info level. Retention was uniform across all indices regardless of value, so debug-level application logs were kept as long as audit records. The storage growth curve had no relationship to business growth, but nobody had the field-level data to explain it.
Environment
On-premises ELK cluster ingesting application, infrastructure, and audit logs from across the business.
Approach
The assessment measures storage consumption at the field level, not just the index level, and cross-references that against search and dashboard usage. That combination shows precisely which fields cost the most and return the least, which turns retention from a policy argument into an evidence-based decision.
Solution
- 1Storage attributed at field level per index pattern using disk usage analysis rather than index totals alone
- 2Field usage cross-referenced against saved searches, dashboards, and query history to find indexed-but-never-searched fields
- 3Dynamic mapping behavior reviewed per pattern, identifying where explicit mappings should replace it
- 4Large payload logging identified at source, with recommendations to sample or truncate request and response bodies
- 5Retention tiers proposed by data class so debug logs and audit records stop sharing a retention policy
- 6Projected storage model showing the growth curve before and after the recommended changes
Outcome
The assessment showed that a large share of indexed fields had never appeared in any query. With explicit mappings and tiered retention, projected storage growth flattened enough to cancel the next planned expansion.
Technologies
ElasticsearchKibanaLogstashRelated Use Cases
ELK Stack Pipeline Architecture Redesign
Consulting engagement to redesign an ELK ingest architecture around buffered queues, ingest node pipelines, and schema standardization.
ELK Stack Pipeline and Ingest Support
Ongoing support across the full ELK ingest path — Beats, Logstash, ingest pipelines, and index templates — with 24/7 coverage.
Ready for a ELK Stack Health Check?
AceMQ's senior ELK Stack engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.