Luigi

API rate limit exceeded due to uncontrolled task concurrency

warning
Connection ManagementUpdated Dec 16, 2016(via Exa)
Technologies:
How to detect:

Luigi tasks calling rate-limited external APIs will exceed API quotas when multiple tasks run concurrently, causing API throttling or task failures when tasks do not declare resource constraints to limit concurrent execution.

Recommended action:

Declare the API call as a resource. Set resource availability in Luigi config [resources] section and declare resource consumption in the task class. This limits concurrent execution to n tasks at a time based on configured resource availability.