Multi-year metric history without multi-year block storage bills
PrometheusThanosAmazon S3
Grafana
KubernetesProvisioned block storage per instance dropped by roughly an order of magnitude, and year-range capacity queries that used to time out now return in seconds against downsampled data. Retention is now…
Overview
A healthcare analytics provider needed long capacity-planning history and multi-year retention for certain operational metrics, but was achieving it by attaching very large block volumes to Prometheus. AceMQ designed an object-storage-backed architecture with downsampling instead.
Challenge
Local retention was set to well over a year, which meant enormous provisioned volumes per instance, slow compaction, and painful instance moves. Queries over long ranges timed out because they scanned raw-resolution samples across the whole window. There was also no real durability story — the retention lived on volumes attached to single instances, so an instance loss meant losing history that the business had committed to keeping.
Environment
Prometheus on Kubernetes across multiple AWS accounts, object storage available, regulated retention obligations on a subset of metrics.
Approach
We separated the two jobs Prometheus was being asked to do: recent high-resolution data for alerting, and long-range history for planning and audit. Prometheus keeps a short local window; blocks ship to object storage where a downsampling compactor produces coarser resolutions for long-range queries. The retention obligation is then enforced by object lifecycle policy rather than instance configuration.
Solution
- 1Local Prometheus retention shortened to a small window sized for alerting and immediate debugging only
- 2Block shipping to object storage with a compactor producing five-minute and one-hour downsampled resolutions
- 3Query layer configured to select the appropriate resolution automatically based on the requested range
- 4Retention obligations enforced through object lifecycle rules per bucket prefix, auditable independently of Prometheus
- 5Deduplication configured across highly available Prometheus pairs so long-range queries return one coherent series
- 6Cost model comparing provisioned block storage against object storage plus downsampling at the customer's real data volumes
Outcome
Provisioned block storage per instance dropped by roughly an order of magnitude, and year-range capacity queries that used to time out now return in seconds against downsampled data. Retention is now enforceable and auditable without depending on any individual instance surviving.
Technologies
PrometheusThanosAmazon S3GrafanaKubernetesRelated Use Cases
Prometheus Scrape and Federation Architecture Assessment
Assessment of a Prometheus federation topology that had grown past its limits, causing gaps and duplicated data across sites.
Prometheus Restart and WAL Replay Support
Ongoing support for large Prometheus instances where restarts caused extended monitoring blind spots due to slow write-ahead log replay.
Grafana Dashboard Estate Assessment
Assessment of a sprawling Grafana dashboard estate to identify duplication, broken panels, and the small set of dashboards anyone actually uses.
Grafana Observability Stack Consolidation
Consulting engagement to consolidate fragmented metrics, logs, and traces onto a single Grafana-based observability layer with consistent labeling.
Grafana Outage and Datasource Timeout Remediation
Remediation of a Grafana deployment that became unusable during incidents, with dashboards timing out exactly when engineers needed them most.
MongoDB Replica Set and Oplog Window Support
Ongoing support for replica sets where a short oplog window was forcing repeated full initial syncs of secondaries during nightly batch loads.
Need Prometheus Architecture Guidance?
AceMQ's senior Prometheus engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.