Emergency Incident Response
Ingest stopped, queries failing, replicas diverged, or a disk filling faster than merges can free it. A senior engineer joins a live bridge within 15 minutes with access to diagnose — not a ticket acknowledgement.
ClickHouse Support
Last reviewed by Scott Sternloff
AceMQ supports ClickHouse in production — "too many parts" from unbatched inserts, memory limits hit on a single join, replication queues stalled behind Keeper, and mutations that rewrite terabytes to change one column. Self-managed or ClickHouse Cloud. Every ticket reaches a named senior engineer who knows your table engines.
Planning a build, upgrade or migration instead? See AceMQ ClickHouse consulting and services.
Customers




Partners


Incident Triage
These are real symptoms from real ClickHouse production environments — and the first thing our engineers check when one comes in.
| Symptom you're seeing | What we check first | Typical resolution |
|---|---|---|
| Inserts failing with DB::Exception: Too many parts | Insert batch size and frequency first — every insert statement creates at least one part per partition, and background merges cannot keep up with thousands of tiny inserts per second. We also check partition key cardinality, because partitioning by day is fine and partitioning by timestamp is not. | Under 1 hour |
| Query killed with "Memory limit exceeded" on a GROUP BY or JOIN | The join algorithm and which side is the build side. ClickHouse loads the right-hand table into memory for a hash join, so a large table on the right will exhaust the limit regardless of settings. Then max_bytes_before_external_group_by, so aggregation spills instead of failing. | 1–2 hours |
| A filtered query still scans the whole table | Whether the WHERE column is a prefix of the ORDER BY key. The primary index is a sparse index over the sorting key, so filtering on the second column without the first gives no pruning at all. Fix is a data skipping index, a projection, or a corrected sorting key. | Under 2 hours |
| Replicas diverging, replication queue growing and not draining | ClickHouse Keeper or ZooKeeper latency and session expiries, then system.replication_queue for a single entry that keeps failing and blocking everything behind it. Keeper problems present as ClickHouse problems, and undersized Keeper nodes are the usual root cause. | 2–4 hours |
| Disk filling despite TTL rules and deletes being configured | system.parts for unmerged parts, system.mutations for a long-running rewrite, and whether TTL moves to cold storage are running at all. Merges and TTL both need free space to work, so a nearly full disk stops the process that would free it. | 1–3 hours |
| An ALTER TABLE UPDATE or DELETE has been running for days | system.mutations parts_to_do. Mutations rewrite every affected part rather than updating rows in place, so a single column change on a large table means rewriting the whole dataset. We move these to lightweight deletes, a ReplacingMergeTree pattern, or a partition-scoped rewrite. | 1–2 hours |
| Distributed queries far slower than the same query on one shard | Shard skew from the sharding key, then whether the query fans out and re-aggregates a huge intermediate result on the initiator node. Pushing the GROUP BY down and returning aggregate states instead of raw rows usually collapses the gap. | 2–4 hours |
| Insert throughput dropped sharply after adding a materialized view | How many materialized views now fire per insert block and what each one writes. MVs run synchronously in the insert path, so the slowest target table sets your ingest rate — and each target has its own merge load competing for the same background pool. | Under 2 hours |
Resolution times reflect typical ClickHouse engagements under an active AceMQ support contract. Every P1 closes with a written root-cause analysis.
What's Included
No add-on pricing for incidents. No per-ticket charges. One contract covers the whole surface.
Ingest stopped, queries failing, replicas diverged, or a disk filling faster than merges can free it. A senior engineer joins a live bridge within 15 minutes with access to diagnose — not a ticket acknowledgement.
Every P1 closes with a written RCA: what failed, why, the fix applied, and the schema, ingest, or settings change that stops it recurring. Delivered as standard, not on request.
Sorting key and primary index design against your real filter patterns, partition granularity, codec and compression selection per column, and choosing between ReplacingMergeTree, AggregatingMergeTree, and plain MergeTree deliberately.
EXPLAIN and query log analysis, join reordering and algorithm selection, projections and skip indexes, plus insert batching or async_insert configuration sized against your producer's actual behaviour.
ReplicatedMergeTree topology, ClickHouse Keeper sizing and latency, replication queue monitoring, and distributed table design that doesn't create hot shards or expensive fan-out.
Version upgrades with a tested rollback, self-managed to ClickHouse Cloud moves in either direction, tiered S3 storage rollout, and migrations onto ClickHouse from Elasticsearch, Druid, or a row-store warehouse.
Why AceMQ
The same senior engineers stay on your account. They know your table engines, your sorting keys, and your ingest pattern — so a P1 call starts with a hypothesis rather than a walkthrough of your schema.
You reach a senior ClickHouse engineer directly by phone, email, or Slack. Nobody collects information to pass along, and no escalation approval stands between you and someone who can read system.query_log.
Most ClickHouse incidents trace back to a sorting key that doesn't match the query filters or a partition key with far too much cardinality. Settings changes can buy time; we tell you when the real fix is a table rebuild and plan the migration.
"Too many parts" is a producer problem that manifests in the database. We tune the Kafka table engine, batching in your ingest service, and async_insert together, because fixing only the server side just moves the failure later.
Engineers across 26+ countries and every time zone. Your 3am ingest stall is someone's mid-afternoon — no overnight skeleton crew, no waiting for a region to come online.
We diagnose across Keeper, disk and object-store latency, the ingest pipeline, network policy, and Kubernetes. ClickHouse is fast enough that the bottleneck is frequently somewhere else entirely.
Cloud, Kubernetes, bare metal, hybrid, and air-gapped — including environments where you can't give us outbound network access.
Escalation Path
Every tier reaches a senior ClickHouse engineer. There is no tier-1 triage layer to get through.
P1 escalation path
A replay of one P1, from the first page to the written root-cause analysis
15-minute P1 response, 24/7.
A response time, not a resolution time.
P1 opened, on-call engineer paged
Phone, email or Slack reaches the on-call senior engineer directly, whether the incident is in RabbitMQ, Kafka, Redis, PostgreSQL or Kubernetes. No web form, no tier-1 queue.
Senior engineer on your bridge
A senior engineer who already knows your environment joins a live bridge, so nobody has to re-explain your topology.
Specialists across the stack join
When an incident crosses components, the engineers who know each one join. For RabbitMQ, the path runs to the core team.
Written root-cause analysis
Root cause, the fix applied and the prevention steps, delivered after every P1.
Pricing
AceMQ ClickHouse support contracts typically run $24,000–72,000 a year, depending on SLA level, scope (number of clusters or nodes, and environments) and deployment (self-managed, cloud-managed or Kubernetes).
| What moves the price | How it changes the contract |
|---|---|
| SLA level | Business-hours cover costs less than 24/7 cover with a 15-minute emergency response. |
| Scope | More clusters or nodes, and more environments, make a larger contract. |
| Deployment | Self-managed, cloud-managed and Kubernetes estates need different amounts of hands-on work. |
Covering several technologies under one contract, for example RabbitMQ, Kafka and Redis, earns an aggressive discount.
Support In Practice
ClickHouse support and remediation work from AceMQ. Each card links to the full write-up: the cluster, what AceMQ diagnosed or changed, and the result.
Adtech Measurement Platform
Resolving ingestion failures where frequent small inserts produced parts faster than background merges could retire them, tripping the parts limit.
Cybersecurity Analytics Provider
Ongoing support for ReplicatedMergeTree clusters covering replication queue stalls, memory limit failures on large queries, and mutation backlogs.
Financial Market Data Provider
Assessment of shard and replica topology, storage tiering, query concurrency limits, and merge behavior ahead of a significant data volume increase.
Industrial IoT Platform Operator
Redesigning ORDER BY keys, partitioning, codecs, and materialized views so dashboard queries read a small fraction of the data instead of full scans.
Support options
AceMQ offers ClickHouse support in four tiers, each handing more of the day-to-day work to AceMQ: 24/7 support and incident management for clusters your team runs, proactive monitoring that catches part counts, mutation debt and Keeper latency first, managed ClickHouse on your own infrastructure, and fully managed ClickHouse on AceMQ infrastructure. Self-managed clusters and ClickHouse Cloud are covered, on any release.
You run it, we back you up
Best for: Teams that run ClickHouse themselves and need senior cover when ingest stops at 3am.
You run it, we watch it with you
Best for: Teams that want the merge backlog or the runaway mutation found in a review, not when inserts start failing.
We run it in your cloud or data centre
Best for: Regulated and air-gapped estates that must keep analytics data inside their own boundary.
We host it and run it, as a service
Best for: Teams that want ClickHouse analytics without a cluster to run, and do not need it in their own account.
| Who does what | Tier 1 24/7 support | Tier 2 Proactive monitoring | Tier 3 Managed, your infrastructure | Tier 4 Fully managed by AceMQ |
|---|---|---|---|---|
| Who runs ClickHouse day to day? | Your team | Your team, with AceMQ reviews | AceMQ | AceMQ |
| Where does it run? | Your infrastructure | Your infrastructure | Your cloud account or data centre | AceMQ infrastructure |
| 24/7 incident response | 15-minute P1 response | 15-minute P1 response | 15-minute P1 response | 15-minute P1 response |
| Health checks and CVE alerts | On request | Scheduled | Included | Included |
| Patching and upgrades | Your team, AceMQ advises | Planned with AceMQ | Done by AceMQ | Done by AceMQ |
| Monitoring and on-call | Your team | Your team | AceMQ | AceMQ |
| Data, keys and accounts | Yours | Yours | Yours | Your data, hosted by AceMQ |
Every Tier can start as an ad hoc engagement or an annual partnership, and you can move up or down a Tier as your team changes.
One partner for ClickHouse
Support is one part of what AceMQ does for ClickHouse. The same engineers deliver consulting, health checks, migrations and end-of-life cover, and support the systems that usually run next to it.
ClickHouse support covers 24/7 emergency incident response with a 15-minute SLA, root cause analysis on every P1, schema and table engine review, query and ingest tuning, replication and Keeper health checks, CVE and patch advisory, quarterly reviews, and upgrade or migration support. Every ticket goes to a named senior engineer assigned to your account. There is no tier-1 triage layer and no per-incident charge.
That error means background merges have fallen behind the rate at which new parts are created, and it is almost always caused by insert shape rather than server capacity. Every insert statement produces at least one part per partition it touches, so an application inserting single rows thousands of times a second creates parts far faster than merges can consolidate them. The fix is batching at the producer — tens of thousands of rows per insert is normal — or enabling async_insert so the server buffers on your behalf. If the partition key has high cardinality, say partitioning by hour or by a user id, each insert hits many partitions and multiplies the problem, so that usually needs correcting too. Raising the parts limit is a stopgap that trades an error today for a slower cluster tomorrow.
P1 emergencies — ingest stopped, queries failing cluster-wide, replicas diverged, disks full — get a 15-minute first response with a senior engineer joining a live bridge. P2 critical issues are 1 hour, P3 high-priority 4 hours, and P4 standard questions next business day. These are contractual response times covering nights, weekends, and holidays.
Yes, with the same engineers. ClickHouse Cloud removes Keeper management, replication mechanics, and storage provisioning, which are genuine operational wins. What it does not remove is schema design, sorting key selection, join memory behaviour, mutation cost, and the insert batching discipline that keeps part counts sane — and those account for the large majority of tickets we see either way. We also help teams decide honestly whether the Cloud compute model is cheaper for their workload than running their own nodes, which depends heavily on query concurrency and idle time.
ClickHouse does not have per-column indexes in the way a row store does. The primary index is a sparse index over the ORDER BY key, and it can only skip granules when the filter matches a prefix of that key. If your sorting key is (tenant_id, event_time) and you filter only on event_time, no pruning happens and you read the whole table. There are three ways out: add a data skipping index on the column if its values are well correlated with the sort order, create a projection with a different sort order, or rebuild the table with a sorting key matching the dominant query shape. We measure which one wins on your data rather than guessing, since skip indexes are frequently less effective than their documentation suggests.
Yes, mutations can be killed with KILL MUTATION and the already-rewritten parts remain valid — but the more useful conversation is why the mutation was needed at all. ClickHouse mutations are not row updates; they rewrite every part that contains matching rows, so changing one column on a large table means rewriting the entire dataset with full merge and compression cost. For deletes, lightweight deletes mark rows instead and are dramatically cheaper. For corrections and upserts, a ReplacingMergeTree or a versioned table with FINAL at query time avoids rewrites entirely. Where a genuine rewrite is unavoidable, we scope it to specific partitions so it runs in bounded chunks.
ClickHouse, Inc. runs a support program for ClickHouse Cloud users and for self-managed customers with an agreement, under its support services policy: support cases are classified by severity level, initial response times depend on the tier, and only recent releases are covered, the current and two prior stable releases or the current and previous long term support release. It is good product support, and it excludes modified builds, third-party hosting and unsupported versions. AceMQ is a support service for the cluster you actually run: any version, any host, with ClickHouse experts who look at the schema, the insert pattern and the queries as well as the server. Most support incidents on ClickHouse are operational rather than product defects, which is why teams hold both.
Replication lag on ReplicatedMergeTree tables usually traces to ClickHouse Keeper or ZooKeeper being slow or overloaded, a replica that cannot fetch a part because the source is busy, or a mutation stuck ahead of everything else in the replication queue. Merge backlog comes from inserts arriving as too many small parts, background pool settings that are too small for the workload, or disks that cannot keep up with merge IO. The fixes are in that order: relieve the coordination service, batch the inserts or enable asynchronous inserts, and size the merge pools and storage for real-time analytics at your ingest rate. The system tables tell the story quickly if you know which ones to read, and we do that on the first call.
Yes. AceMQ works on ClickHouse in two ways: ad hoc engagements for a defined piece of work, such as a health check, an upgrade or an incident, or an annual partner engagement that covers 24/7 support, proactive reviews and named engineers. Many teams start ad hoc and move to an annual partnership once the first engagement is done.
Every AceMQ proposal is tailored to your requirements: your ClickHouse environment, the SLA you need, the hours of cover and the scope of work. Most contracts run $24,000–72,000 a year, depending on the SLA, the scope and how ClickHouse is deployed, with aggressive discounts when several technologies share one contract. Request a quote and AceMQ replies within 1 hour.
Let's talk
Reactive cover, proactive reviews, or AceMQ running ClickHouse for you on your infrastructure or ours. A senior engineer scopes it with you, and quote requests are answered within 1 hour.
Production incident right now?Email a ClickHouse Architect
Send us a message and we'll follow up within 1 hour — or book a free 30-min consultation directly.
305-204-2607info@acemq.comPrefer to talk now? Call us directly or use the consultation tab to find a time that works.