Multiple workers can overload system without global resource limits
warningResource ContentionUpdated Apr 6, 2016(via Exa)
Technologies:
How to detect:
When running multiple Luigi workers, the sum of resources required by concurrent tasks can overload the machine. This occurs without central scheduler resource enforcement configured.
Recommended action:
Configure Luigi's central scheduler resource limits to enforce global resource constraints across all workers. See the resources configuration section in Luigi documentation. Consider porting resource logic to worker side for farm deployments as simpler alternative to frameworks like Mesos.