Pick the executor that matches your task profile, not the one in the tutorial

Celery and Kubernetes executors have opposite cost profiles. Celery keeps warm workers, which suits many short tasks and wastes capacity on few long ones; Kubernetes creates a pod per task, which suits heterogeneous long-running tasks and adds meaningful startup overhead to short ones. AceMQ measures the task profile and designs accordingly.

Apache Airflow 2.x on Kubernetes, with mixed short-running orchestration tasks and long-running compute tasks submitted to Spark, Databricks, or dbt.

AceMQ profiles task duration, concurrency, and resource footprint distribution across the deployment to determine which executor fits, and whether a split — Kubernetes executor with a Celery queue for high-frequency short tasks — is justified. The migration is then staged by DAG group with both executors available, and log, secret, and dependency handling is validated before cutover.

• 1
  Task duration, concurrency, and resource footprint profiling to determine executor fit against measured behavior
• 2
  Executor recommendation including hybrid designs where short and long task classes justify separate queues
• 3
  Pod template, resource request, and node selector design for Kubernetes executor tasks with per-task-class overrides
• 4
  Worker autoscaling and queue routing design for Celery deployments, sized against measured concurrency
• 5
  Remote log persistence, secrets backend, and image build pipeline aligned to the target executor
• 6
  Staged DAG-group migration with rollback, running both executors in parallel until each group is validated