Back to all use cases

Managed ServicesSoftware / Digital Platforms

Airflow stays in your account and your DAGs stay yours, but the pager is not

SaaS Platform Team

What a Tier 3 managed Airflow engagement covers when a platform team wants Airflow run for it inside its own cloud account, with DAG ownership staying in-house.

Overview

A typical request comes from a data platform team that owns many DAGs but does not want to own the scheduler, the metadata database or the upgrade calendar. AceMQ operates standard Apache Airflow inside the customer's own cloud account and Kubernetes cluster, with system access, under the Tier 3 managed contract.

Challenge

Airflow's platform layer fails in ways that look like DAG problems: a metadata database that grows without a retention rule, a scheduler loop slowed by DAG parse time, tasks stuck in queued because a pool or queue name does not match, and provider packages that drift between environments. Nobody on the platform team has time to watch these, and the move from Airflow 2 to Airflow 3 is a project on top.

Environment

Apache Airflow on Kubernetes using the official Helm chart, with CeleryExecutor or KubernetesExecutor, a PostgreSQL metadata database and a secrets backend, in the customer's own cloud account.

Approach

AceMQ agrees where the line sits: AceMQ owns the platform, the customer owns DAG code. It reviews the current deployment, writes the operating runbooks, takes over monitoring and on-call, and plans version upgrades against the Apache project's published support dates.

Solution

  1. Scheduler, DAG processor, worker and triggerer operations on the customer's Kubernetes cluster, with executor and pool sizing reviewed against measured load
  2. Metadata database care: connection and table-growth monitoring, a retention rule run with airflow db clean, backups and restore tests
  3. Monitoring and on-call for scheduler heartbeat, DAG parse time, tasks stuck in queued, pool slot use and failed runs
  4. Provider and dependency pinning with constraint files, and each upgrade rehearsed on a copy before it is run in an agreed window
  5. An Airflow 3 upgrade plan that lists the DAGs using removed features, such as SubDAGs, SLAs and direct database access from tasks
  6. Triage of DAG-level causes, such as top-level code and pool saturation, reported to the DAG owners with the file named

Outcome

The customer ends up with Airflow run to a written runbook inside its own account, an upgrade calendar tied to the project's support dates, and a clear line between what AceMQ operates and what the DAG teams own. Runbooks and deployment code sit in the customer's repository.

Technologies

Apache AirflowKubernetesPostgreSQLCelery
  • Assessment

    Apache Airflow Platform Assessment

    Regional Utility OperatorEnergy

    Reviewing an Airflow deployment for reliability, DAG authoring practice, secrets handling, and upgrade readiness before it becomes unmaintainable.

    Apache AirflowKubernetesPostgreSQLApache Spark
    Read case study: Apache Airflow Platform Assessment
  • Consulting

    Apache Airflow Executor Migration and Platform Design

    Healthcare Analytics ProviderHealthcare

    Moving from Celery to the Kubernetes executor, or the reverse, with a sizing model and deployment design that matches the workload profile.

    Apache AirflowKubernetesCeleryApache Spark
    Read case study: Apache Airflow Executor Migration and Platform Design
  • Support

    Apache Airflow DAG Parse Time Support

    Digital Retail PlatformSoftware

    Fixing scheduler delay caused by DAG files that make network or database calls at parse time, blocking every DAG in the deployment.

    Apache AirflowKubernetesPostgreSQL
    Read case study: Apache Airflow DAG Parse Time Support
  • Consulting

    Docker Image Build and Layer Caching Optimization

    Enterprise Software VendorSoftware

    Restructuring Dockerfiles and CI caching so builds reuse layers properly, cutting pipeline time and image size across a large service estate.

    DockerKubernetesKongPostgreSQL
    Read case study: Docker Image Build and Layer Caching Optimization
  • Remediation

    Kubernetes Failed Rollout and Rollback Remediation

    Subscription Software ProviderSoftware

    Replacing a release process that kept rolling out broken builds with dependency-aware probes, automated abort and a rehearsed rollback.

    KubernetesHelmArgo RolloutsPostgreSQL
    Read case study: Kubernetes Failed Rollout and Rollback Remediation

Related Reading

Looking for Apache Airflow Managed Operations?

AceMQ's senior Apache Airflow engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.