24/7 ClickHouse support with senior engineers and a 15-minute emergency SLA
Kubernetes
PrometheusReplication stalls are now detected and cleared before they affect query results, and memory-related query failures dropped to a small fraction of their previous rate. The customer's team resolves mos…
Overview
A security analytics provider ran ReplicatedMergeTree tables across a multi-shard cluster. Replication queues would stall behind stuck entries, ad-hoc analyst queries hit memory limits and failed, and delete mutations accumulated without completing. AceMQ took over support with named senior engineers and no tier-1 triage.
Challenge
ClickHouse failure modes are unfamiliar to teams coming from row-store databases. A stalled replication queue entry blocks everything behind it and needs deliberate intervention. Memory limit failures on analyst queries were being answered by raising limits, which pushed the cluster toward host-level memory pressure instead. Mutations were being issued as routine deletes without regard for the rewrite cost they impose.
Environment
ClickHouse on Kubernetes with ClickHouse Keeper coordination, serving security event search and analytics.
Approach
AceMQ established diagnosis procedures for each recurring failure class so the customer's team could handle common cases directly, reserving escalation for genuinely novel incidents. Query-level controls replaced blanket memory limit increases, and mutation practice was brought under policy.
Solution
- 1Built diagnosis and recovery procedures for stalled replication queue entries and Keeper session issues
- 2Replaced blanket memory limit increases with per-query and per-user limits plus controlled spilling to disk
- 3Rewrote the highest-cost analyst queries to use projections and aggregate tables instead of full scans
- 4Established mutation policy covering batching, scheduling, and when TTL-based expiry is the better tool
- 5Instrumented replication delay, mutation backlog, and memory usage per query as tracked signals
- 6Provided 24/7 escalation directly to senior ClickHouse engineers under a 15-minute emergency SLA
Outcome
Replication stalls are now detected and cleared before they affect query results, and memory-related query failures dropped to a small fraction of their previous rate. The customer's team resolves most recurring cases from the runbooks without escalating.
Technologies
KubernetesPrometheusRelated Use Cases
ClickHouse Too Many Parts Remediation
Resolving ingestion failures where frequent small inserts produced parts faster than background merges could retire them, tripping the parts limit.
ClickHouse Cluster and Storage Assessment
Assessment of shard and replica topology, storage tiering, query concurrency limits, and merge behavior ahead of a significant data volume increase.
Need Expert ClickHouse Support?
AceMQ's senior ClickHouse engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.