Knowing where the cluster runs out before the data arrives

Overview

A market data provider was onboarding additional exchange feeds that would multiply stored volume. Rather than scale the cluster proportionally, they engaged AceMQ to determine which parts genuinely needed to grow and which were misconfigured.

Challenge

Scaling ClickHouse well requires knowing which resource actually binds. Merge throughput, query concurrency, Keeper coordination load, and storage bandwidth each fail differently and are frequently confused for one another. The existing cluster had never been profiled under sustained load, so no one knew which would bind first.

Environment

ClickHouse across on-premises hardware and cloud object storage, serving historical and intraday market data queries.

Approach

AceMQ profiled the cluster under representative sustained load rather than synthetic benchmarks, tracking merge throughput, query queueing, Keeper request latency, and storage bandwidth simultaneously to see which saturated first. Findings were mapped to the projected volume increase.

Solution

• Profiled the cluster under representative sustained load to identify which resource saturates first
• Reviewed shard and replica topology against query routing and failure tolerance requirements
• Assessed Keeper deployment, request latency, and coordination load as a scaling constraint in its own right
• Evaluated tiered storage configuration and the movement of aged parts to object storage
• Reviewed query concurrency limits, queueing behavior, and per-user resource controls
• Delivered a capacity model tying projected data volume to specific scaling actions and their sequence

Outcome

Technologies