Size the bookie tier first — that is where Pulsar deployments go wrong
KubernetesCustomers get tier-by-tier sizing derived from the requirement that actually constrains each one, rather than a single node count scaled from message rate. Existing deployments usually find that at le…
Overview
Pulsar separates serving from storage, which is the source of both its operational flexibility and its sizing complexity. Broker capacity, bookie capacity, and metadata capacity scale on different inputs, and most undersized deployments got the bookie tier wrong. AceMQ sizes each tier against measured requirements.
Challenge
Sizing exercises that focus on message rate miss the constraints that actually bind. Write quorum multiplies raw storage by the replication factor, so retention drives bookie disk far faster than expected. Journal devices need low-latency fsync rather than capacity. Topic and partition counts drive ZooKeeper or etcd load and broker memory independently of throughput. Geo-replication adds cross-region bandwidth that is rarely budgeted.
Environment
New or expanding Apache Pulsar deployments on Kubernetes, bare metal, or hybrid infrastructure, with tiered offload to object storage.
Approach
AceMQ derives requirements from measured or projected throughput, retention, durability, and topic cardinality, then sizes each tier against the input that constrains it. Where the deployment already exists, current utilization is profiled to separate genuine capacity limits from configuration problems that more hardware would not fix.
Solution
- 1Broker tier sizing against connection count, topic cardinality, dispatch throughput, and cache working set
- 2Bookie tier sizing with write quorum and ack quorum applied to retention, separating journal and ledger device requirements
- 3Metadata store sizing and configuration for ZooKeeper or etcd against topic, partition, and ledger counts
- 4Tiered offload thresholds and object storage sizing for long-retention topics
- 5Geo-replication topology and cross-region bandwidth modeling where multi-region delivery is required
- 6Failure-domain and rack-awareness placement so ensemble selection survives a zone or rack loss
Outcome
Customers get tier-by-tier sizing derived from the requirement that actually constrains each one, rather than a single node count scaled from message rate. Existing deployments usually find that at least one reported capacity problem is a configuration issue that additional hardware would not have solved.
Technologies
KubernetesRelated Use Cases
Apache Pulsar Multi-Tenancy Design
Designing tenant, namespace, and policy structure so independent teams can share a Pulsar cluster without interfering with each other.
Apache Pulsar BookKeeper IO Remediation
Resolving cluster-wide write latency caused by bookie journal and ledger device contention, and restoring write quorum headroom.
Ready for a Apache Pulsar Health Check?
AceMQ's senior Apache Pulsar engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.