BlogElasticsearch

Elasticsearch Heap, Shards and Cluster Health: Operations Guide

Scott Sternloff

By Scott Sternloff, Senior Enterprise Architect

LinkedIn · Updated · 4 min read

On this page

Short answer

Most Elasticsearch incidents trace to three numbers: how much heap a node has, how many shards it carries, and how full its disks are. Keep heap at or below half of memory and under the compressed-pointer limit, keep shards between 10GB and 50GB and under 1,000 per node, and keep disks below the 85% watermark. This guide gives the checks and API calls behind each. It is for teams who run their own clusters and want to know when to call for Elasticsearch support.

Most Elasticsearch incidents trace to three numbers: how much heap a node has, how many shards it carries, and how full its disks are. Keep heap at or below half of memory and under the compressed-pointer limit, keep shards between 10GB and 50GB and under 1,000 per node, and keep disks below the 85% watermark. This guide gives the checks and API calls behind each. It is for teams who run their own clusters and want to know when to call for Elasticsearch support.

How much heap should an Elasticsearch node have?

Set the minimum and maximum heap (Xms and Xmx) to the same value, keep it at or below 50% of the memory available to the node, and keep it under the compressed ordinary object pointers threshold. Elastic's JVM settings page says 26GB is safe on most systems and up to 30GB on some. Elastic also notes that Elasticsearch sets heap automatically from each node's roles and memory and recommends the default sizing for most production environments, so override it only with a reason.

Verify, do not assume:

  • The startup log line reads like heap size [1.9gb], compressed ordinary object pointers [true].
  • GET _nodes?filter_path=**.jvm.using_compressed_ordinary_object_pointers shows the same.
  • GET _cat/nodes?v&h=name,heap.percent,heap.max,ram.percent shows pressure per node.

Heap that stays high after garbage collection is the warning sign, not a spike during a query.

What do the circuit breakers tell you?

They tell you which request type is eating heap. Defaults from the circuit breaker settings page:

BreakerSettingDefault
Parentindices.breaker.total.limit95% of heap with real-memory accounting (70% without)
Fielddataindices.breaker.fielddata.limit40% of heap
Requestindices.breaker.request.limit60% of heap
In-flight requestsnetwork.breaker.inflight_requests.limit100% of heap, bounded by the parent

GET _nodes/stats/breaker shows trip counts per node. Fielddata trips point at sorting or aggregating on a text field; request trips point at a large aggregation. Raising a limit moves the failure, so fix the query or mapping first.

How many shards is too many?

Elastic's sizing guidance is to target shards of 10GB to 50GB, keep each shard below 200 million documents, and keep non-frozen shards to no more than 1,000 per node by default. For dedicated master nodes, keep fewer than 3,000 indices per GB of master heap: four-gigabyte heaps support fewer than 12,000 indices.

SymptomLikely causeCheck
Master CPU high, cluster state updates slowThousands of tiny shardsGET _cluster/stats (total shards, mapping size)
Search slow with flat data volumeMany small shards searched per queryGET _cat/shards?v&s=store
Heap high on data nodesMapping overhead per shardtotal_estimated_overhead in _nodes/stats
Empty indices piling upRollover on max_age with no dataGET _cat/count/<pattern>

Fix with index lifecycle management: set max_primary_shard_size to 50gb and min_primary_shard_size to 10gb on rollover, and use _shrink, _split or _reindex for indices already sized wrong. cluster.max_shards_per_node can be raised temporarily, and Elastic's page says to reset it to null once the long-term fix is in. To avoid hotspots, index.routing.allocation.total_shards_per_node caps shards per node per index.

Why is the cluster yellow or red?

Yellow means replicas are unassigned; red means at least one primary is. Ask the cluster, one shard at a time:

`` GET _cluster/health?level=indices GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason POST _cluster/allocation/explain ``

The allocation explain API reports why a shard is unassigned, or why it stays where it is, with the decision of each allocation decider. The most common answer is disk. Elastic's allocation settings page gives these defaults:

WatermarkDefault (disk used)Effect
Low85%No new shards allocated to the node
High90%Elasticsearch tries to move shards off the node
Flood stage95%index.blocks.read_only_allow_delete applied to indices on the node

The flood-stage block is removed automatically once usage falls below the high watermark. Free space first; raising watermarks is a stopgap.

What should you check every week?

CheckCallHealthy sign
Heap after GC_cat/nodes?h=name,heap.percentNot trending up
Breaker trips_nodes/stats/breakerZero or flat
Shard size_cat/shards?s=store:descLargest under 50GB, most over 10GB
Shards per node_cat/allocation?vUnder 1,000
Disk_cat/allocation?vBelow 85%
Empty indices_cat/countNone
ILMGET <index>/_ilm/explainNo index stuck on a step
SnapshotsGET _snapshot/_all/_allA recent success you have restored from

Test the restore. A snapshot that reports success has not proved you can recover.

When to call for help

Call when the cluster is red and allocation/explain points at something you cannot change, when heap stays above the compressed-pointer limit or breakers keep tripping after a query fix, or when shard counts have outgrown the cluster state. AceMQ provides 24/7 Elasticsearch and OpenSearch support across the Americas, EMEA and APAC, with response targets from 15 minutes, by plan, and quote replies within 1 hour. Support typically runs $24,000-72,000 a year (USD), by SLA, scope and deployment. See also who offers Elasticsearch support and Elasticsearch pricing. Many of these API names also exist in OpenSearch; check that project's documentation for defaults.

Sources

  • Elastic docs, size your shards: elastic.co/docs/deploy-manage/production-guidance/optimize-performance/size-shards
  • Elastic docs, JVM settings: elastic.co/docs/reference/elasticsearch/jvm-settings
  • Elastic docs, circuit breaker settings: elastic.co/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings
  • Elastic docs, cluster-level shard allocation and routing settings: elastic.co/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings
  • Elastic API docs, cluster allocation explain: elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-allocation-explain

Frequently Asked Questions

What is the right Elasticsearch heap size?

Equal `Xms` and `Xmx`, at most 50% of node memory and under about 26GB (up to 30GB on some systems), per Elastic's JVM settings page. The default automatic sizing is recommended for most production use.

How many shards per node is safe?

Elastic's default limit is 1,000 non-frozen shards per node. Target 10GB to 50GB per shard.

Why did my indices turn read-only?

A node passed the 95% flood-stage disk watermark, so `index.blocks.read_only_allow_delete` was applied. It lifts when usage drops below 90%.

What does the cluster allocation explain API do?

It explains why a shard is unassigned or has not moved, including each allocation decider's decision.

What causes circuit breaker exceptions?

Requests that would push heap past a breaker limit: parent 95%, fielddata 40%, request 60% by default. Fix the query or mapping before changing limits.

Scott Sternloff

Author

Scott SternloffSenior Enterprise Architect

Senior Enterprise Architect at AceMQ. Writes the technical coverage here: RabbitMQ, Kafka and Redis architecture, clustering and migrations.

Free Consultation

Get Expert Eyes on Your Elasticsearch Environment

Whether you're troubleshooting a production incident, planning a migration, or want a second opinion on your architecture — our team is ready. No pitch, just answers.

Email Us