Scheduler retry-delay parameter requires local-scheduler flag to take effect
warningconfigurationUpdated Sep 15, 2021(via Exa)
Technologies:
How to detect:
The --scheduler-retry-delay CLI parameter is ignored when using a separately started luigid central scheduler, causing tasks to retry on the default 15-minute interval instead of the specified delay. This affects external task dependency checks and can cause unnecessarily long wait times between retries.
Recommended action:
Add --local-scheduler flag to the luigi command (e.g., 'python -m luigi --module <module> <task> --scheduler-retry-delay 1 --local-scheduler') to enable custom retry delays. Alternatively, configure the retry-delay parameter before starting the luigid scheduler separately, as it cannot be changed after luigid is already running.