The scheduler is running, the DAGs are unpaused, and nothing is being queued

Overview

When Airflow stops scheduling but the scheduler process is healthy, the usual cause is that tasks marked running no longer have a live process behind them. Those tasks hold executor slots and pool capacity indefinitely, and once the slots are exhausted nothing new enters the queue. AceMQ clears the condition and removes the cause.

Challenge

Zombie tasks accumulate from worker pods evicted mid-task, Celery workers killed by the OOM killer, and network partitions that break the heartbeat without updating task state. The metadata database still shows them running, so the executor counts them against parallelism and pool slots. Meanwhile scheduler loop time degrades as the task instance table grows, which slows detection further. Restarting the scheduler does not help, because the stale state is in the database.

Environment

Apache Airflow 2.x on Kubernetes or VMs with Celery or Kubernetes executors and a PostgreSQL or MySQL metadata database.

Approach

AceMQ identifies the stuck task instances and reconciles their state against actual worker processes, freeing the held slots so scheduling resumes. Zombie detection thresholds and heartbeat timeouts are then aligned with real task runtimes, worker resource limits are corrected where eviction was the trigger, and metadata database maintenance is put on a schedule so scheduler loop time stays bounded.

Solution

• Identification and reconciliation of stuck task instances against live worker processes to release executor and pool slots
• Zombie detection threshold and task heartbeat interval aligned with real long-running task durations
• Worker resource limit and eviction review where pod termination is the source of orphaned tasks
• Pool, parallelism, dag_concurrency, and max_active_tasks configuration corrected against actual worker capacity
• Metadata database maintenance including task instance and log table cleanup with a retention policy
• Scheduler health alerting on queued-task age and scheduler loop duration so the next stall is detected in minutes

Outcome

Technologies