Scheduler blocks tasks when resource limits are exceeded
warningResource ContentionUpdated Mar 8, 2017(via Exa)
Technologies:
How to detect:
The Luigi scheduler prevents new tasks from running when configured resource limits would be exceeded, causing tasks to remain in PENDING state until resources are released by completing tasks
Recommended action:
Monitor resource usage metrics (luigi.resources.used vs luigi.resources.total). If tasks are frequently blocked, either increase resource limits in [resources] configuration or reduce concurrent worker task limits. Check scheduler pending/queue metrics to identify resource bottlenecks