Back to all use cases

Architecture AdvisorySoftware / Digital Platforms

Decide what the observability stack is made of, and who is on call for it, before the next team adds its own

Platform Engineering Team

What a stack review covers when metrics, logs and dashboards grew one tool at a time and nobody owns the limits, the retention or the pager.

Overview

A typical review starts when several teams each run their own Prometheus, a log cluster has grown past its owner and a dashboard estate has no map. AceMQ reads what runs today, what each team needs from it and what the organisation can operate, and returns a written stack decision and operating model.

Challenge

Each team chose for its own services: one Prometheus per cluster with local retention only, a search cluster holding logs with shards sized by default, and dashboards that query three different sources under different label names. No one owns a label policy, retention is whatever the disk allows, alerts fire in batches, and the platforms that carry the messaging and databases are monitored in a separate place.

Environment

Several Kubernetes clusters and a set of virtual machines, a Prometheus server in each, one OpenSearch cluster for logs, Grafana in front of both, and RabbitMQ, Kafka and PostgreSQL as the platforms underneath the services.

Approach

AceMQ builds an inventory of sources, tools and owners, measures series and log volume by source, and compares the options against it: Prometheus alone or with Thanos or Mimir for metrics, OpenSearch or Loki for logs, Tempo or Jaeger for traces, with OpenTelemetry in the collection layer. The review ends in a decision record and an agreed operating model.

Solution

  1. Inventory of every collector, store and dashboard source, with an owner and a cost to run for each
  2. A label policy and series limits, with the rule for who may add a label and where it is enforced
  3. Retention sized from the sample rate and ingest volume, with the point where Thanos or Mimir becomes necessary stated in numbers from the estate
  4. A shard and index lifecycle plan for the log cluster, including snapshots and a restore test
  5. RabbitMQ, Kafka and PostgreSQL signals added to the same stack, with alerts routed to the team that owns each platform
  6. An operating model that names who holds the pager for the stack itself and which of the four support tiers applies

Outcome

The customer ends up with one written stack decision instead of several tool choices, a label and retention policy its engineers have reviewed, an alert routing design and an agreed owner for each layer, including the monitoring stack itself.

Technologies

PrometheusGrafanaOpenSearchKubernetes
  • Consulting

    Grafana Observability Stack Consolidation

    Financial Services GroupFinancial Services

    Consulting engagement to consolidate fragmented metrics, logs, and traces onto a single Grafana-based observability layer with consistent labeling.

    GrafanaPrometheusLokiTempoKubernetes
    Read case study: Grafana Observability Stack Consolidation
  • Consulting

    Prometheus Long-Term Storage and Downsampling Design

    Healthcare Analytics ProviderHealthcare

    Consulting engagement to design multi-year Prometheus metric retention with downsampling and object storage, replacing oversized local disks.

    PrometheusThanosAmazon S3GrafanaKubernetes
    Read case study: Prometheus Long-Term Storage and Downsampling Design
  • Remediation

    Grafana Outage and Datasource Timeout Remediation

    Telecommunications OperatorTelecommunications

    Remediation of a Grafana deployment that became unusable during incidents, with dashboards timing out exactly when engineers needed them most.

    GrafanaPrometheusKubernetesPostgreSQL
    Read case study: Grafana Outage and Datasource Timeout Remediation
  • Managed Services

    Managed Grafana and Prometheus in the Customer's Kubernetes Cluster

    Platform Engineering TeamCross-Industry

    A typical engagement for a platform team that hands day-to-day operation of Prometheus, Alertmanager and Grafana to AceMQ, with everything running in the team's own Kubernetes cluster.

    GrafanaPrometheusAlertmanagerKubernetes
    Read case study: Managed Grafana and Prometheus in the Customer's Kubernetes Cluster

Related Reading

Have a Prometheus Challenge Like This?

AceMQ's senior Prometheus engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.