Back to all use cases

SupportCross-Industry

Finding out why a shard has no leader, and making sure it does not happen again

Content Publisher

A typical emergency engagement: a SolrCloud shard that stopped accepting updates after a node loss, diagnosed from cluster state, replica types and ZooKeeper, with a written root-cause analysis.

Overview

A typical engagement starts when a search cluster loses a node and one shard stops accepting updates while queries are still answered. The on-call team can see replicas in a recovering state but cannot tell why no leader is elected.

Challenge

A SolrCloud shard needs a leader chosen from its replicas, and a PULL replica is never eligible. A layout built for read scaling can leave a single leader-eligible replica per shard, so one node failure is enough to stop indexing. ZooKeeper health, garbage-collection pauses that outlast the client timeout and the update log all affect how quickly a shard recovers.

Environment

SolrCloud on Kubernetes with a ZooKeeper ensemble, collections using a mix of NRT, TLOG and PULL replicas, and an indexing pipeline that retries on failure.

Approach

AceMQ engineers join a live bridge, read the cluster state and the node logs, and check which replicas are leader-eligible and whether ZooKeeper has kept its quorum. They restore an eligible replica first and look for the cause only once indexing has resumed, then review replica types, ZooKeeper timeouts and heap behavior.

Solution

  1. Cluster state and replica type review for each affected shard to confirm which replicas can become leader
  2. ZooKeeper member health and latency check, and a comparison of Solr garbage-collection pauses with the client timeout
  3. Restoration of a leader-eligible replica so the shard accepts updates again
  4. Replica layout change so that every shard keeps more than one leader-eligible replica
  5. ZooKeeper and Solr timeout settings reviewed against observed pause times
  6. Written runbook for the next node loss, so the on-call team can check eligibility and quorum without escalating

Outcome

The engagement delivers the restored shard, a written root-cause analysis covering the replica layout and the timeout settings, the layout and configuration changes that prevent a repeat, and a runbook the customer's on-call team can follow for the same condition.

Technologies

Apache SolrSolrCloudZooKeeperKubernetes
  • Consulting

    Solr 8 to Solr 10 Upgrade Planning

    Software Platform TeamSoftware

    A typical engagement for a team running Solr 8 in production: a written upgrade path to Solr 10 that names every intermediate version, every reindex and every removed feature, rehearsed on a copy first.

    Apache SolrSolrCloudZooKeeperJava
    Read case study: Solr 8 to Solr 10 Upgrade Planning
  • Support

    OpenSearch Cluster Support for Regulated Workloads

    Public Sector AgencyGovernment

    24/7 enterprise support for OpenSearch clusters carrying regulated search and audit workloads, including security plugin and upgrade coverage.

    OpenSearchOpenSearch DashboardsLinuxKubernetes
    Read case study: OpenSearch Cluster Support for Regulated Workloads
  • Support

    ClickHouse MergeTree and Replication Support

    Cybersecurity Analytics ProviderCross-Industry

    Ongoing support for ReplicatedMergeTree clusters covering replication queue stalls, memory limit failures on large queries, and mutation backlogs.

    ClickHouseClickHouse KeeperKubernetesPrometheus
    Read case study: ClickHouse MergeTree and Replication Support
  • Support

    Docker Disk Exhaustion and Build Cache Support

    Media Streaming CompanyCross-Industry

    Stopping recurring build agent and node outages caused by unpruned Docker build cache, dangling images, and orphaned volumes filling the filesystem.

    DockerKubernetesAirbytePostgreSQL
    Read case study: Docker Disk Exhaustion and Build Cache Support
  • Remediation

    Apache Pulsar BookKeeper IO Remediation

    Global Payments ProcessorFinancial Services

    Resolving cluster-wide write latency caused by bookie journal and ledger device contention, and restoring write quorum headroom.

    Apache PulsarApache BookKeeperKubernetesZooKeeper
    Read case study: Apache Pulsar BookKeeper IO Remediation
  • Consulting

    Apache Pulsar Multi-Tenancy Design

    Managed Services ProviderSoftware

    Designing tenant, namespace, and policy structure so independent teams can share a Pulsar cluster without interfering with each other.

    Apache PulsarApache BookKeeperKubernetesZooKeeper
    Read case study: Apache Pulsar Multi-Tenancy Design

Related Reading

Need Expert Apache Solr Support?

AceMQ's senior Apache Solr engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.