Observability support

Prometheus Support for Kubernetes, Thanos and Mimir Estates

Last reviewed by Tyler Eastridge

AceMQ supports Prometheus 24/7 across Americas, EMEA and APAC: Kubernetes with the Prometheus Operator, bare metal, Thanos, Mimir and managed Prometheus services. The calls we get most are an out-of-memory kill from label cardinality, a restart that takes an hour in WAL replay, and alerts that fire but never page anyone.

24/7 Prometheus support with response targets from 15 minutes, by plan. Quote replies within 1 hour.

Support quotes answered within 1 hour, by a senior engineer.

Planning a build, upgrade or migration instead? See AceMQ Prometheus consulting and services.

160+
Customers supported
10+
Years in production support
40+
Sev 1 incidents handled a year
24/7
Americas, EMEA and APAC
1hour
Quote reply

Customers

  • Microsoft
  • Henry Schein
  • Enbridge
  • Gojek

Partners

  • Broadcom
  • Redis
  • VMware Tanzu
  • OSSeva

Incident Triage

Problems we fix every week

Failures behind most Prometheus pages we take, with the metric, endpoint or amtool command an engineer reaches for first.

Common Prometheus production problems and how AceMQ resolves them
Symptom you're seeingWhat we check first
Memory and series count climb togetherHead memory follows prometheus_tsdb_head_series. We open /api/v1/status/tsdb for the series count per metric and label, run topk(10, count by (__name__)({__name__=~".+"})) to find the worst metrics, and compare scrape_series_added per job to see which target started producing new series. The permanent fix is metric_relabel_configs with labeldrop for labels that carry IDs or paths, and sample_limit and label_limit on each scrape job so one bad exporter cannot take the server down.
Restart takes an hour or moreStartup time is WAL replay plus loading memory-mapped chunks. We read prometheus_tsdb_data_replay_duration_seconds and prometheus_tsdb_wal_corruptions_total, and for estates that restart often we enable --enable-feature=memory-snapshot-on-shutdown so replay starts from a snapshot. A faster disk helps less than fewer series.
Targets show up == 0 or flapWe compare scrape_duration_seconds with scrape_timeout, check prometheus_target_scrapes_exceeded_sample_limit_total, and read the target's error text. Common causes are a scrape timeout longer than the interval, a TLS certificate that expired, and a ServiceMonitor selector that stopped matching after a label change.
Recording and alerting rules fall behindWhen prometheus_rule_group_last_duration_seconds exceeds the group interval, or prometheus_rule_group_iterations_missed_total rises, alerts arrive late. We find the expensive expression, move it into a recording rule, and split large rule groups so they evaluate in parallel.
Alerts fire and nobody is pagedWe run amtool config routes test against the real label set, list active silences with amtool silence query, and read alertmanager_notifications_failed_total by integration. On HA pairs we check alertmanager_cluster_members so gossip is not splitting notifications or dropping them.
remote_write drops samplesWe watch prometheus_remote_storage_samples_pending, prometheus_remote_storage_shards against max_shards, and prometheus_remote_storage_samples_dropped_total, then tune queue capacity, max_samples_per_send and write relabeling to send less.
Thanos queries time out or the compactor stopsthanos_compact_halted set to 1 means the compactor stopped on overlapping or corrupted blocks and downsampling stops with it. We repair the block, restart safely, and check that store gateways have index caches sized for the query range.
Disk fills ahead of retentionRetention bounds time unless --storage.tsdb.retention.size is set. We compare prometheus_tsdb_storage_blocks_bytes with churn from short-lived pods, which create fresh series every restart.

Findings end with the label, job or rule responsible and the change that keeps it from returning.

Prometheus stuck replaying its WAL and alerts silent? Request a quote and we reply in under an hour.

Contact Us

Deployment coverage

Which Prometheus deployments does AceMQ support, and who covers what?

DeploymentWhat AceMQ coversWhat the vendor coversExcluded
Prometheus Operator and kube-prometheus-stack on KubernetesServer, rules, Alertmanager, ServiceMonitors, sharding, upgrade planningKubernetes distribution vendor supports the platformCluster control-plane faults
Prometheus on VMs, bare metal or air-gappedServer, storage sizing, federation, HA pairs, upgrades without outbound accessNo vendor; the open-source project has no support contractHardware and OS faults outside the agreed scope
Thanos or Mimir for long-term storageCompactor, store gateway, query path, object storage layout, block repairOpen-source projects with community channels; a commercial vendor only if you buy oneObject storage platform faults
Cloud-managed Prometheus servicesScrape configuration, cardinality, recording rules, alert design, cost driversThe cloud provider runs the service and its serversProvider platform faults; limits set by the service
Alertmanager HA clustersRouting trees, inhibition, silences, notification integrations, gossipReceiver vendors (paging, chat) support their own toolsThird-party notification outages

Support options

Who provides enterprise support for Prometheus? Project, cloud vendors and independent support compared

Prometheus is a Cloud Native Computing Foundation project with no vendor support contract. Enterprise support comes from a cloud or observability vendor, if you buy their managed service, or from an independent provider for the Prometheus you run.

Prometheus project / Cloud-managed Prometheus
AceMQ independent support
ChannelsPrometheus project: Slack on CNCF, Matrix, prometheus-users mailing list, Discourse forum, GitHub (Prometheus community page, read 17 Nov 2026)Cloud-managed Prometheus: Provider support for the service
ChannelsPhone and email, one contract
Response commitmentPrometheus project: NoneCloud-managed Prometheus: Set by the cloud support plan
Response commitmentResponse targets from 15 minutes, by plan, 24/7
Patch policyPrometheus project: A new minor release every 6 weeks; minor releases generally stop getting bug fixes after that; LTS releases get selected fixes for one year, with high-severity (CVSS 7.0 or higher) issues guaranteed, on a best-effort community basis (Prometheus release cycle page, read 17 Nov 2026)Cloud-managed Prometheus: Per the service
Patch policyWe diagnose any version and plan the move to a current line
Cardinality and costPrometheus project: Not in scopeCloud-managed Prometheus: You pay for what you ingest; the service does not fix your labels
Cardinality and costIn scope
PricePrometheus project: FreeCloud-managed Prometheus: Usage-based
Price$24,000 to $72,000 a year (USD), set by SLA, scope and deployment

What's Included

Everything in your Prometheus support contract

Prometheus servers, Alertmanager, recording rules and remote storage such as Thanos or Mimir, all included with no extra charge per call.

  • 01

    Emergency Incident Response

    Prometheus down, alerting silent, or a server stuck in WAL replay while your on-call is blind. With response targets from 15 minutes, by plan, a senior engineer joins a live bridge — monitoring being down during an incident is a P1 on its own.

  • 02

    Cardinality Control

    Finding the metric and label that caused the growth, dropping it at the right layer with metric_relabel_configs, and putting guardrails in the exporter or instrumentation so the next well-meaning commit does not repeat it.

  • 03

    PromQL & Recording Rule Tuning

    Rewriting queries that scan too many series, building the recording rules your dashboards should have had, and setting evaluation intervals that keep rule groups finishing well inside their window.

  • 04

    Alertmanager Design

    Routing trees, grouping, inhibition, silences, and high-availability Alertmanager clusters. We prune alerts that never led to action and make sure the ones that survive reach the right team once.

  • 05

    Long-Term Storage Architecture

    Thanos, Mimir, and Cortex — sidecar versus receive, compactor and downsampling health, object store layout, and query fan-out. We fix the compaction failures that make historical queries slow long before anyone notices.

  • 06

    Federation, Sharding & Scale

    When to shard by scrape target rather than federate, how to keep hierarchical federation from becoming a single point of failure, and remote_write tuning for multi-cluster and multi-region estates.

Why AceMQ

Why monitoring teams keep us on call

  • Named Engineers, Zero Cold Start

    The same senior engineers stay on your account. They know your scrape topology, your series budget, and your alert routing — so a P1 call opens with diagnosis instead of you describing the setup.

  • No Tier-1 Triage Layer

    You reach a senior engineer directly by phone, email, or Slack. Nobody collects details to pass along, and there is no escalation approval standing between you and the person who can fix it.

  • Cardinality Is the Job, Not a Footnote

    Most Prometheus incidents are one metric with one bad label. We find it fast, drop it at the correct layer, and fix the instrumentation — rather than doubling the memory limit and waiting for the same page in six weeks.

  • The Whole Stack, Including Grafana

    Prometheus almost never runs alone. We support the Grafana on top, the Thanos or Mimir behind it, the exporters feeding it, and the Kubernetes underneath — so nothing gets handed back as somebody else's layer.

  • Genuine Follow-the-Sun Coverage

    Round-the-clock coverage for customers in 26+ countries. Your 3am page is someone's mid-afternoon — no overnight skeleton crew, no waiting for a region to come online.

  • Proactive, Not Just Reactive

    Quarterly reviews of series growth, rule evaluation timing, and storage headroom, so you resize ahead of the OOM rather than during it. Cardinality problems announce themselves weeks before they page anyone.

Prometheus setups we look after

Operator-managed clusters, OpenShift monitoring, cloud-managed services fed by remote_write, and isolated networks with no outbound access.

  • Kubernetes (Prometheus Operator, kube-prometheus-stack)
  • OpenShift monitoring stack
  • AWS (EC2, EKS, Amazon Managed Prometheus)
  • Microsoft Azure (AKS, Azure Monitor managed)
  • Google Cloud (GKE, Google Managed Prometheus)
  • Thanos, Mimir & Cortex
  • VictoriaMetrics
  • VMware vSphere & Tanzu
  • Bare metal & on-premise
  • Air-gapped / no outbound access
  • Alertmanager HA clusters
  • Grafana Cloud remote_write

Version coverage

Versions we support

Versions we support
LineReleaseProject support window (release cycle page, 17 Nov 2026)
Prometheus 3.131 July 2026Project support through 31 July 2027
Prometheus 3.x (earlier minors)VariousEach minor receives fixes for about 6 weeks
Prometheus 2.xVariousWe support these in production and plan the move to 3.x

The project designates long-term releases (2.37, 2.45, 2.53, 3.5, 3.13 on the release cycle page). Thanos, Mimir, Alertmanager and the Operator are supported at whatever version you run.

Escalation Path

Your first hour of a Prometheus outage

Silent alerting in the middle of an outage is the worst case, so the fastest clock is reserved for it.

  1. From 15 min (by plan)first responseEmergencyPrometheus down or stuck in WAL replay, Alertmanager not delivering while production is degraded, or an HA pair both failing
  2. 1 hourfirst responseCriticalRule groups running minutes behind, remote_write dropping samples, Thanos compactor halted on bad blocks
  3. 4 hoursfirst responseHighDashboards timing out over 30-day ranges, a flapping target, disk filling ahead of retention
  4. Next Dayfirst responseStandardRecording rule design, routing tree review, retention sizing, Thanos or Mimir selection

Targets vary by plan. A silent Alertmanager during an outage gets the shortest clock, a retention question the longest.

Prefer an engineer who already knows your scrape jobs and routing tree? Request a quote or message us.

Contact Us

Pricing

What does AceMQ Prometheus support cost?

Prometheus support is typically $24,000–72,000 a year. The price tracks how many servers and active series you run, whether Thanos, Mimir or federation sit on top, how many Alertmanager receivers page people, and your response level.

Prices are quoted in US dollars (USD). Response targets start at 15 minutes by plan, 24/7, and quote requests are answered within 1 hour.

Support In Practice

Prometheus problems we have resolved

Typical Prometheus support engagements. Each card links to the scenario: the metrics estate, what AceMQ checks or changes, and what the engagement delivers.

1 / 6 case studies

All use cases

Support options

Four tiers of Prometheus support.

AceMQ offers Prometheus support in four tiers, each handing more of the day-to-day work to AceMQ: 24/7 support and incident management for servers your team runs, proactive monitoring that catches cardinality and alert-routing problems first, managed Prometheus on your own infrastructure, and fully managed Prometheus on AceMQ infrastructure. The 3.x LTS lines and 2.x are covered in all four.

  • Tier 1

    Prometheus 24/7 Support & Incident Management

    You run Prometheus, we answer escalations

    Best for: SRE teams with their own servers and rules who want a senior engineer when memory, rules or paging fail.

  • Tier 2

    Prometheus 24/7 Support with Proactive Monitoring

    Cardinality and alert reviews on a schedule

    Best for: Teams who see series count climbing or alert noise growing and want it audited each quarter.

  • Tier 3

    Managed Prometheus on Your Infrastructure

    We operate Prometheus and Alertmanager in your estate

    Best for: Organizations running Thanos, Mimir or federation in their own clusters without enough hands to maintain it.

  • Tier 4

    Fully Managed Prometheus on AceMQ Infrastructure

    We run the monitoring stack as a service

    Best for: Companies that want metrics, rules and paging working without owning Prometheus operations.

FAQ

Prometheus support questions

Still have questions about Prometheus? Email an Expert

Who provides enterprise support for Prometheus?

The Prometheus project sells no support, and managed-service vendors cover only their own service. For servers you run, AceMQ answers 24/7, with response from 15 minutes by plan. A Sev 1 such as silent alerting means an engineer on a bridge, testing your routing tree with amtool.

Why is Prometheus using so much memory?

Memory follows active series. Look at prometheus_tsdb_head_series and the TSDB status page for the metrics and labels creating series, then drop high-cardinality labels at scrape time. More memory only delays the next crash and lengthens the next restart.

How long should a Prometheus restart take?

Minutes on a healthy server. Hour-long restarts mean WAL replay over a large head. Fewer series and the memory snapshot on shutdown feature shorten it.

How do we set up high availability for Prometheus?

Run two identical servers that scrape the same targets, send alerts from both to an Alertmanager cluster, and put Thanos or Mimir in front for a single query view. HA covers server loss; it does not deduplicate rule load or fix cardinality.

Thanos or Mimir: which one fits?

Thanos suits estates that already run a Prometheus per cluster and want a global query view over object storage. Mimir suits higher write volume with a central write path. We size either against your series count, retention and query pattern.

Why do alerts not reach the on-call engineer?

Test the real label set against the routing tree with amtool config routes test, check for old silences and inhibit rules, and read notification failure counters per integration.

Can you reduce our alert noise?

Yes. We count which alerts fired and led to action over a quarter, remove the rest, then tune for durations, grouping and inhibition on what stays. On-call teams usually feel the difference within a couple of weeks.

What does a Prometheus health check include?

We measure series growth per job, scrape health, rule group duration, Alertmanager routing and clustering, disk against retention, and remote_write queues. The output is a written list of fixes ordered by risk, with the query behind each finding.

Is there a way to try Prometheus support without a year-long deal?

Yes. Hire us for a cardinality audit, an Alertmanager routing rebuild or a Thanos design and stop at the deliverable. Teams that value overnight cover and engineers who remember their scrape jobs tend to convert that work into an annual agreement.

Why does Prometheus support cost what it does?

Most contracts fall between $24,000 and $72,000 a year (USD). The inputs are Prometheus server and cluster count, active series, any Thanos, Mimir or remote_write layer, Alertmanager complexity and the response level. Send a short description and we quote within 1 hour.

Let's talk

Monitoring That Fails During an Incident Makes Every Incident Longer

Paste your /api/v1/status/tsdb output into a message. We will name the labels that are costing you memory and say what to drop.

A senior Prometheus engineer, not a sales queueQuote requests answered within 1 hourAd hoc engagements or an annual partnership

Production incident right now?Email a Prometheus Architect