Back to all use cases
Financial Services / PaymentsRemediationCloud / Kubernetes

In Pulsar, one struggling bookie raises write latency for the entire cluster

GP
Global Payments Processor

Overview

Pulsar brokers acknowledge a write once the ack quorum of bookies confirms it, so the slowest bookie in each ensemble sets the latency floor. A single node with a saturated journal device degrades write latency across every topic whose ledgers touch it. AceMQ isolates the bookie and fixes the storage layout underneath.

Challenge

The usual root cause is journal and ledger data sharing a device. The journal needs low-latency sequential fsync; the ledger device does large sequential writes and periodic compaction. Together on one disk they contend, journal fsync latency climbs, and add-entry latency rises cluster-wide. Contributing factors include an ensemble and quorum configuration with no headroom for a slow node, an undersized write cache, and garbage collection on the entry log competing with live traffic.

Environment

Apache Pulsar with BookKeeper on Kubernetes or bare metal, ZooKeeper or etcd metadata, and tiered offload to object storage.

Approach

AceMQ correlates per-bookie journal fsync and add-entry latency against broker-side publish latency to identify which node is setting the floor, then confirms whether the cause is device contention, cache sizing, or compaction. The immediate action isolates the bookie from new ensembles; the durable fix separates journal and ledger devices and retunes cache and compaction.

Solution

  • Per-bookie journal fsync and add-entry latency correlated with broker publish latency to isolate the constraining node
  • Journal and ledger device separation with appropriate storage classes and independent IOPS budgets
  • Write cache, read-ahead cache, and direct memory sizing tuned against actual working set and JVM heap
  • Entry log garbage collection and compaction thresholds adjusted so reclamation stops competing with live writes
  • Ensemble, write quorum, and ack quorum configuration set to tolerate a slow node without stalling writes
  • Autorecovery and bookie replacement procedures validated, including under-replicated ledger recovery

Outcome

Cluster-wide publish latency returns to its baseline once the constraining bookie is isolated and the storage layout corrected, and the revised quorum configuration means the next slow node degrades gracefully rather than stalling the cluster.

Technologies

Apache PulsarApache BookKeeperKubernetesZooKeeper

Ready to Get Started?

Whether you need architecture advisory, 24/7 support, or full managed services, AceMQ has the expertise to help.

Contact Us