Resource requirements block task scheduling despite available workers
criticalResource ContentionUpdated Feb 1, 2018(via Exa)
Sources
Technologies:
How to detect:
Tasks requiring CPU resources (e.g., 10 cores) remain pending indefinitely when Luigi worker is not configured with resource parameters. Workers report 'no more tasks to run' while showing pending tasks unique to the worker. Workflow stalls completely despite no failed tasks.
Recommended action:
Configure resource parameters in luigi.cfg under [resources] section (e.g., cpu_cores=10) or pass resource parameter when launching workflow (e.g., {'cpu_cores': 10}). Match or exceed the CPU core requirements of your lowest-level tasks. Alternatively, set resources attribute on individual Task classes to control concurrency (e.g., resources = {'max_workers': 1}).