Finding out which fields you are indexing that nobody ever searches

Overview

An insurance services provider's ELK cluster had outgrown its storage twice in eighteen months with no clear reason. AceMQ assessed what was actually being indexed and how much of it anyone used.

Challenge

Default dynamic mapping meant every field in every JSON log line was both indexed and stored, including large nested structures nobody had ever searched. Several services logged full request and response bodies at info level. Retention was uniform across all indices regardless of value, so debug-level application logs were kept as long as audit records. The storage growth curve had no relationship to business growth, but nobody had the field-level data to explain it.

Environment

On-premises ELK cluster ingesting application, infrastructure, and audit logs from across the business.

Approach

The assessment measures storage consumption at the field level, not just the index level, and cross-references that against search and dashboard usage. That combination shows precisely which fields cost the most and return the least, which turns retention from a policy argument into an evidence-based decision.

Solution

• Storage attributed at field level per index pattern using disk usage analysis rather than index totals alone
• Field usage cross-referenced against saved searches, dashboards, and query history to find indexed-but-never-searched fields
• Dynamic mapping behavior reviewed per pattern, identifying where explicit mappings should replace it
• Large payload logging identified at source, with recommendations to sample or truncate request and response bodies
• Retention tiers proposed by data class so debug logs and audit records stop sharing a retention policy
• Projected storage model showing the growth curve before and after the recommended changes

Outcome

Technologies