Luigi

Unspecified resources default to limit 1, causing unexpected task serialization

warning
Resource ContentionUpdated Mar 8, 2017(via Exa)
Technologies:
How to detect:

When a task declares resource usage but no corresponding limit is configured in the [resources] section, Luigi defaults to a limit of 1, forcing tasks requiring that resource to run serially instead of in parallel

Recommended action:

Explicitly configure all resource limits in the [resources] configuration section. Add entries like 'hive: 2' or 'mysql: 1' for each resource type used by tasks. Review task resource declarations and ensure matching configuration entries exist with appropriate concurrency limits