Size the bookie tier first — that is where Pulsar deployments go wrong

Overview

Pulsar separates serving from storage, which is the source of both its operational flexibility and its sizing complexity. Broker capacity, bookie capacity, and metadata capacity scale on different inputs, and most undersized deployments got the bookie tier wrong. AceMQ sizes each tier against measured requirements.

Challenge

Sizing exercises that focus on message rate miss the constraints that actually bind. Write quorum multiplies raw storage by the replication factor, so retention drives bookie disk far faster than expected. Journal devices need low-latency fsync rather than capacity. Topic and partition counts drive ZooKeeper or etcd load and broker memory independently of throughput. Geo-replication adds cross-region bandwidth that is rarely budgeted.

Environment

New or expanding Apache Pulsar deployments on Kubernetes, bare metal, or hybrid infrastructure, with tiered offload to object storage.

Approach

AceMQ derives requirements from measured or projected throughput, retention, durability, and topic cardinality, then sizes each tier against the input that constrains it. Where the deployment already exists, current utilization is profiled to separate genuine capacity limits from configuration problems that more hardware would not fix.

Solution

• Broker tier sizing against connection count, topic cardinality, dispatch throughput, and cache working set
• Bookie tier sizing with write quorum and ack quorum applied to retention, separating journal and ledger device requirements
• Metadata store sizing and configuration for ZooKeeper or etcd against topic, partition, and ledger counts
• Tiered offload thresholds and object storage sizing for long-retention topics
• Geo-replication topology and cross-region bandwidth modeling where multi-region delivery is required
• Failure-domain and rack-awareness placement so ensemble selection survives a zone or rack loss

Outcome

Technologies