Luigi

Scheduler blocks tasks when resource limits are exceeded

warning
Resource ContentionUpdated Mar 8, 2017(via Exa)
Technologies:
How to detect:

The Luigi scheduler prevents new tasks from running when configured resource limits would be exceeded, causing tasks to remain in PENDING state until resources are released by completing tasks

Recommended action:

Monitor resource usage metrics (luigi.resources.used vs luigi.resources.total). If tasks are frequently blocked, either increase resource limits in [resources] configuration or reduce concurrent worker task limits. Check scheduler pending/queue metrics to identify resource bottlenecks