Back to all use cases

RemediationSoftware / Digital Platforms

The rollout reported success while customers saw errors

Subscription Software Provider

Replacing a release process that kept rolling out broken builds with dependency-aware probes, automated abort and a rehearsed rollback.

Overview

Kubernetes replaces pods as fast as the readiness probe allows, and it does not roll back a stalled Deployment by itself. A probe that only checks that a port is open lets a broken build replace healthy pods, and the rollback then depends on someone knowing which revision to return to.

Challenge

Two releases in one quarter shipped a build that started cleanly and failed on its first database call. Probes passed, so Helm reported success and the Deployments replaced every replica before the alerts fired. Rolling back meant a manual Helm rollback by whoever was on call, and the Deployments kept only two old ReplicaSets.

Environment

A cluster of about 60 services on Google GKE, Helm releases deployed from a CI pipeline, Cloud SQL for PostgreSQL behind most services, and Prometheus alerting.

Approach

AceMQ replayed the two bad releases in a staging namespace to see exactly when each failure became visible, then changed the release path so that a bad build is stopped at the first replica and rolled back without a person having to decide. The changes went in service by service, starting with the three services that carry the most traffic.

Solution

  1. Replaced port-open readiness checks with endpoints that exercise the database and cache connections each service needs
  2. Added startup probes so slow starts no longer fail liveness checks
  3. Set maxUnavailable to 0 with a single surge pod and a progress deadline matched to the measured start time
  4. Moved the highest-traffic services to Argo Rollouts with an analysis step on error rate and p99 latency that aborts and rolls back
  5. Raised revisionHistoryLimit so earlier revisions stay available, and added a rollout undo step to the on-call runbook
  6. Rehearsed a deliberately bad release in staging each month until on-call engineers could roll back from the runbook alone

Outcome

Over the next two quarters one release was aborted automatically at the first replica, and no customer-facing rollout needed a manual rollback. On-call engineers now treat an aborted rollout as routine.

Technologies

KubernetesHelmArgo RolloutsPostgreSQL
  • Consulting

    RabbitMQ Kubernetes Stabilization

    Smart Home Security PlatformCross-Industry

    Hardening RabbitMQ in Kubernetes environments with StatefulSet tuning, quorum queue optimization, storage isolation, and memory/network configuration.

    RabbitMQKubernetes
    Read case study: RabbitMQ Kubernetes Stabilization
  • Remediation

    Docker OOMKilled Container Remediation

    Online Marketplace OperatorSoftware

    Resolving containers repeatedly OOMKilled because the JVM and Node runtimes inside them were sizing heap against host memory rather than the cgroup limit.

    DockerKubernetesPostgreSQLRedis
    Read case study: Docker OOMKilled Container Remediation
  • Remediation

    Kong 502 and Upstream Health Check Remediation

    Digital Commerce PlatformSoftware

    Tracing intermittent 502s at the Kong gateway to misconfigured active health checks and stale DNS resolution of upstream service names.

    KongKubernetesNGINXPostgreSQLDocker
    Read case study: Kong 502 and Upstream Health Check Remediation
  • Consulting

    Docker Image Build and Layer Caching Optimization

    Enterprise Software VendorSoftware

    Restructuring Dockerfiles and CI caching so builds reuse layers properly, cutting pipeline time and image size across a large service estate.

    DockerKubernetesKongPostgreSQL
    Read case study: Docker Image Build and Layer Caching Optimization
  • Support

    Kong Rate Limiting Consistency Support

    B2B SaaS ProviderSoftware

    Fixing rate limits that allowed several times the configured quota because the plugin was using the local counter policy across a multi-node gateway.

    KongRedisKubernetesPostgreSQL
    Read case study: Kong Rate Limiting Consistency Support
  • Remediation

    Airbyte Connector Schema Drift Remediation

    Retail Analytics OrganizationCross-Industry

    Diagnosing an Airbyte connection that kept reporting success while the destination table quietly went stale after an upstream schema change.

    AirbytePostgreSQLSnowflakeKubernetesdbt
    Read case study: Airbyte Connector Schema Drift Remediation

Related Reading

Facing a Kubernetes Production Issue?

AceMQ's senior Kubernetes engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.