Pick the executor that matches your task profile, not the one in the tutorial

Overview

Celery and Kubernetes executors have opposite cost profiles. Celery keeps warm workers, which suits many short tasks and wastes capacity on few long ones; Kubernetes creates a pod per task, which suits heterogeneous long-running tasks and adds meaningful startup overhead to short ones. AceMQ measures the task profile and designs accordingly.

Challenge

Deployments usually inherit an executor choice rather than making one. Celery deployments running thousands of short tasks queue behind a fixed worker pool; Kubernetes executor deployments running the same tasks pay pod startup on every one. Task dependency resolution, log persistence, and secret handling all differ between the two, so a migration touches more than the executor setting. Doing it without a validated rollback risks the entire orchestration layer.

Environment

Apache Airflow 2.x on Kubernetes, with mixed short-running orchestration tasks and long-running compute tasks submitted to Spark, Databricks, or dbt.

Approach

AceMQ profiles task duration, concurrency, and resource footprint distribution across the deployment to determine which executor fits, and whether a split — Kubernetes executor with a Celery queue for high-frequency short tasks — is justified. The migration is then staged by DAG group with both executors available, and log, secret, and dependency handling is validated before cutover.

Solution

• Task duration, concurrency, and resource footprint profiling to determine executor fit against measured behavior
• Executor recommendation including hybrid designs where short and long task classes justify separate queues
• Pod template, resource request, and node selector design for Kubernetes executor tasks with per-task-class overrides
• Worker autoscaling and queue routing design for Celery deployments, sized against measured concurrency
• Remote log persistence, secrets backend, and image build pipeline aligned to the target executor
• Staged DAG-group migration with rollback, running both executors in parallel until each group is validated

Outcome

Technologies