Finding out why a cluster with plenty of hardware still feels slow

Overview

An agency running daily indices for several years had accumulated tens of thousands of small shards. Adding hardware had stopped helping, and master elections were taking long enough to interrupt indexing. AceMQ assessed the cluster to determine whether the problem was capacity or design.

Challenge

Every daily index carried the default shard count regardless of whether it held a few megabytes or hundreds of gigabytes. The resulting cluster state was large enough that publishing an update to all nodes took seconds, pending tasks backed up behind it, and any master failover turned into a multi-minute outage. Nobody could safely change the index template because it was unclear which downstream searches depended on the existing index naming.

Environment

On-premises Elasticsearch cluster, daily time-based indices with multi-year retention, mixed log and document search workloads.

Approach

The assessment inventories shard count and size distribution per index pattern, measures cluster-state size and publication latency, and maps which indices actually receive queries. We separate indices that need to be reshaped from ones that simply need to be rolled up or deleted, and deliver a sequenced plan the customer can execute during normal maintenance.

Solution

• Full shard inventory by index pattern with size distribution, showing which patterns produce shards far below or above useful size
• Cluster-state size and publication latency measured to quantify master overhead directly
• Query traffic mapped per index so retention decisions are made from evidence rather than policy assumption
• Recommended target shard sizing and rollover strategy using ILM with size-based and age-based conditions
• Sequenced remediation plan covering shrink, reindex, and rollup steps in an order that keeps search available throughout
• Capacity model showing what the cluster actually needs after consolidation, avoiding an unnecessary hardware purchase

Outcome

Technologies