Luigi

Resource requirements block task scheduling despite available workers

critical
Resource ContentionUpdated Feb 1, 2018(via Exa)
Technologies:
How to detect:

Tasks requiring CPU resources (e.g., 10 cores) remain pending indefinitely when Luigi worker is not configured with resource parameters. Workers report 'no more tasks to run' while showing pending tasks unique to the worker. Workflow stalls completely despite no failed tasks.

Recommended action:

Configure resource parameters in luigi.cfg under [resources] section (e.g., cpu_cores=10) or pass resource parameter when launching workflow (e.g., {'cpu_cores': 10}). Match or exceed the CPU core requirements of your lowest-level tasks. Alternatively, set resources attribute on individual Task classes to control concurrency (e.g., resources = {'max_workers': 1}).