Knowing where the cluster runs out before the data arrives
PrometheusThe provider onboarded the additional feeds with targeted scaling rather than a proportional cluster expansion. Two constraints that would have bound before storage — Keeper coordination and merge thr…
Overview
A market data provider was onboarding additional exchange feeds that would multiply stored volume. Rather than scale the cluster proportionally, they engaged AceMQ to determine which parts genuinely needed to grow and which were misconfigured.
Challenge
Scaling ClickHouse well requires knowing which resource actually binds. Merge throughput, query concurrency, Keeper coordination load, and storage bandwidth each fail differently and are frequently confused for one another. The existing cluster had never been profiled under sustained load, so no one knew which would bind first.
Environment
ClickHouse across on-premises hardware and cloud object storage, serving historical and intraday market data queries.
Approach
AceMQ profiled the cluster under representative sustained load rather than synthetic benchmarks, tracking merge throughput, query queueing, Keeper request latency, and storage bandwidth simultaneously to see which saturated first. Findings were mapped to the projected volume increase.
Solution
- 1Profiled the cluster under representative sustained load to identify which resource saturates first
- 2Reviewed shard and replica topology against query routing and failure tolerance requirements
- 3Assessed Keeper deployment, request latency, and coordination load as a scaling constraint in its own right
- 4Evaluated tiered storage configuration and the movement of aged parts to object storage
- 5Reviewed query concurrency limits, queueing behavior, and per-user resource controls
- 6Delivered a capacity model tying projected data volume to specific scaling actions and their sequence
Outcome
The provider onboarded the additional feeds with targeted scaling rather than a proportional cluster expansion. Two constraints that would have bound before storage — Keeper coordination and merge throughput — were addressed ahead of the increase.
Technologies
PrometheusRelated Use Cases
ClickHouse Schema and Sort Key Design Consulting
Redesigning ORDER BY keys, partitioning, codecs, and materialized views so dashboard queries read a small fraction of the data instead of full scans.
ClickHouse MergeTree and Replication Support
Ongoing support for ReplicatedMergeTree clusters covering replication queue stalls, memory limit failures on large queries, and mutation backlogs.
Ready for a ClickHouse Health Check?
AceMQ's senior ClickHouse engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.