Finding out why a cluster with plenty of hardware still feels slow
ElasticsearchKibanaLinuxShard count fell dramatically once daily indices were consolidated and old data was rolled up, cutting cluster-state publication time to a fraction of what it was. Master failover became a non-event, …
Overview
An agency running daily indices for several years had accumulated tens of thousands of small shards. Adding hardware had stopped helping, and master elections were taking long enough to interrupt indexing. AceMQ assessed the cluster to determine whether the problem was capacity or design.
Challenge
Every daily index carried the default shard count regardless of whether it held a few megabytes or hundreds of gigabytes. The resulting cluster state was large enough that publishing an update to all nodes took seconds, pending tasks backed up behind it, and any master failover turned into a multi-minute outage. Nobody could safely change the index template because it was unclear which downstream searches depended on the existing index naming.
Environment
On-premises Elasticsearch cluster, daily time-based indices with multi-year retention, mixed log and document search workloads.
Approach
The assessment inventories shard count and size distribution per index pattern, measures cluster-state size and publication latency, and maps which indices actually receive queries. We separate indices that need to be reshaped from ones that simply need to be rolled up or deleted, and deliver a sequenced plan the customer can execute during normal maintenance.
Solution
- 1Full shard inventory by index pattern with size distribution, showing which patterns produce shards far below or above useful size
- 2Cluster-state size and publication latency measured to quantify master overhead directly
- 3Query traffic mapped per index so retention decisions are made from evidence rather than policy assumption
- 4Recommended target shard sizing and rollover strategy using ILM with size-based and age-based conditions
- 5Sequenced remediation plan covering shrink, reindex, and rollup steps in an order that keeps search available throughout
- 6Capacity model showing what the cluster actually needs after consolidation, avoiding an unnecessary hardware purchase
Outcome
Shard count fell dramatically once daily indices were consolidated and old data was rolled up, cutting cluster-state publication time to a fraction of what it was. Master failover became a non-event, and the planned hardware expansion was deferred.
Technologies
ElasticsearchKibanaLinuxRelated Use Cases
Elasticsearch Index Lifecycle and Tiering Design
Consulting engagement to design index lifecycle management, data tiering, and snapshot policy for a regulated Elasticsearch estate.
Elasticsearch Cluster Stuck Yellow After Node Loss
Emergency remediation of an Elasticsearch cluster that would not return to green after a data node failure, with replicas blocked by disk watermarks.
Ready for a Elasticsearch Health Check?
AceMQ's senior Elasticsearch engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.