The scheduler is running, the DAGs are unpaused, and nothing is being queued
KubernetesCelery
PostgreSQLScheduling resumes immediately, and the combination of corrected timeouts, worker sizing, and database maintenance keeps slots from being held by dead tasks. Stalls that previously went unnoticed unti…
Overview
When Airflow stops scheduling but the scheduler process is healthy, the usual cause is that tasks marked running no longer have a live process behind them. Those tasks hold executor slots and pool capacity indefinitely, and once the slots are exhausted nothing new enters the queue. AceMQ clears the condition and removes the cause.
Challenge
Zombie tasks accumulate from worker pods evicted mid-task, Celery workers killed by the OOM killer, and network partitions that break the heartbeat without updating task state. The metadata database still shows them running, so the executor counts them against parallelism and pool slots. Meanwhile scheduler loop time degrades as the task instance table grows, which slows detection further. Restarting the scheduler does not help, because the stale state is in the database.
Environment
Apache Airflow 2.x on Kubernetes or VMs with Celery or Kubernetes executors and a PostgreSQL or MySQL metadata database.
Approach
AceMQ identifies the stuck task instances and reconciles their state against actual worker processes, freeing the held slots so scheduling resumes. Zombie detection thresholds and heartbeat timeouts are then aligned with real task runtimes, worker resource limits are corrected where eviction was the trigger, and metadata database maintenance is put on a schedule so scheduler loop time stays bounded.
Solution
- 1Identification and reconciliation of stuck task instances against live worker processes to release executor and pool slots
- 2Zombie detection threshold and task heartbeat interval aligned with real long-running task durations
- 3Worker resource limit and eviction review where pod termination is the source of orphaned tasks
- 4Pool, parallelism, dag_concurrency, and max_active_tasks configuration corrected against actual worker capacity
- 5Metadata database maintenance including task instance and log table cleanup with a retention policy
- 6Scheduler health alerting on queued-task age and scheduler loop duration so the next stall is detected in minutes
Outcome
Scheduling resumes immediately, and the combination of corrected timeouts, worker sizing, and database maintenance keeps slots from being held by dead tasks. Stalls that previously went unnoticed until a morning report was missing now alert within minutes.
Technologies
KubernetesCeleryPostgreSQLRelated Use Cases
Apache Airflow DAG Parse Time Support
Fixing scheduler delay caused by DAG files that make network or database calls at parse time, blocking every DAG in the deployment.
Apache Airflow Executor Migration and Platform Design
Moving from Celery to the Kubernetes executor, or the reverse, with a sizing model and deployment design that matches the workload profile.
Facing a Apache Airflow Production Issue?
AceMQ's senior Apache Airflow engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.