The Scheduler Parsing CPU Spiral
warningIn Airflow deployments using CeleryExecutor with RabbitMQ as the broker, frequent DAG parsing combined with high Celery task rates can create CPU contention on the RabbitMQ broker, especially when handling task result messages.
Monitor RabbitMQ CPU usage alongside Airflow scheduler activity when using CeleryExecutor. High broker CPU correlating with scheduler heartbeat cycles or task queue depth indicates broker overload from orchestration overhead.
Tune Airflow scheduler settings: increase min_file_process_interval (30s+) to reduce parsing frequency, adjust scheduler_heartbeat_sec based on responsiveness needs. Configure Celery worker concurrency appropriately (16-32). Use Redis instead of RabbitMQ for Celery result backend to reduce broker load. Scale RabbitMQ cluster if task throughput remains high.