Back to all use cases

Managed ServicesSoftware / Digital Platforms

A small platform team owns the cluster calendar, the pager and the product

SaaS Platform Team

A typical Tier 3 engagement: AceMQ takes over upgrades, node pools, certificates, backups and on-call for production EKS clusters that stay in the customer's AWS accounts.

Overview

EKS runs the control plane, but node groups, add-ons, certificates, backups and every upgrade deadline stay with the customer. A small platform team ends up planning around AWS support dates in between product work. Under Tier 3 managed services, AceMQ operates the clusters inside the customer's own AWS accounts.

Challenge

The platform team runs several production clusters and has no time to own the version calendar. Minor upgrades are postponed until AWS extended support is the only option, node groups lag the control plane, add-on versions drift, and nobody has restored a Velero backup since it was configured. Certificates are renewed by whoever notices an expiry date first.

Environment

Production and staging EKS clusters in two AWS accounts, managed node groups across three availability zones, the VPC CNI, CoreDNS and kube-proxy add-ons, ingress and cert-manager, Prometheus alerting, and a GitOps repository for delivery.

Approach

AceMQ starts with an assessment of version position against the EKS support calendar, removed APIs, add-on and operator compatibility, backup and restore history, certificate expiry and monitoring gaps. It remediates the active risks, agrees access and a change process in the customer's accounts, then takes the pager. Cluster platform work is delivered by AceMQ with partner Kubernetes specialists under one contract.

Solution

  1. Assessed each cluster's version against the EKS standard and extended support dates and wrote a minor-by-minor upgrade plan
  2. Scanned manifests, Helm releases and live objects for APIs removed by each target release and added the check to CI
  3. Brought node groups and add-ons in line with the control plane, replacing nodes group by group with pauses sized to measured recovery time
  4. Deployed Velero with retention set against the customer's recovery objectives and scheduled restore tests into a scratch cluster
  5. Inventoried every certificate that can take a cluster down and put expiry alerts and rotation runbooks in place
  6. Took over monitoring and on-call, with the 15-minute Sev 1 response and a written root-cause analysis after each P1

Outcome

The customer receives a version calendar and upgrade plan for every cluster, a tested restore procedure, certificate and add-on inventories, current runbooks and a handover pack. Upgrades, patching and the pager move to AceMQ, and the platform team returns to product work. Clusters, data and the AWS bill stay in the customer's accounts.

Technologies

KubernetesAmazon EKSVelerocert-managerPrometheus
  • Support

    Kubernetes StatefulSet Quorum Loss During Node Drains

    Regional Payments ProcessorFinancial Services

    Stopping a three-member RabbitMQ cluster from losing quorum every time EKS node groups were drained for patching, by fixing readiness, topology spread, disruption budgets and the drain procedure.

    KubernetesRabbitMQAmazon EKSPodDisruptionBudgets
    Read case study: Kubernetes StatefulSet Quorum Loss During Node Drains
  • Remediation

    ClickHouse Too Many Parts Remediation

    Adtech Measurement PlatformSoftware

    Resolving ingestion failures where frequent small inserts produced parts faster than background merges could retire them, tripping the parts limit.

    ClickHouseKafkaKubernetesPrometheus
    Read case study: ClickHouse Too Many Parts Remediation
  • Support

    Tanzu Postgres Checkpoint Stalls on Throttled Volumes

    Software-as-a-Service Analytics VendorSoftware

    Support engagement tracing afternoon write latency spikes on Tanzu Postgres to volume burst credits, not the database settings everyone had been tuning.

    Tanzu PostgresPostgreSQLKubernetesPrometheus
    Read case study: Tanzu Postgres Checkpoint Stalls on Throttled Volumes

Related Reading

Looking for Kubernetes Managed Operations?

AceMQ's senior Kubernetes engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.