Technologies/Luigi/luigi.scheduler.pending_tasks
LuigiLuigiMetric

luigi.scheduler.pending_tasks

Tasks waiting for execution
Dimensions:None
Available on:PrometheusPrometheus (1)DatadogDatadog (1)
Interface Metrics (2)
PrometheusPrometheus
Current number of tasks in PENDING state waiting for dependencies or resources
Dimensions:None
DatadogDatadog
Number of tasks pending execution
Dimensions:None
Sources
luigi_scheduler_pending_tasksgithub.com
luigi.scheduler.pending_tasksgithub.com

Technical Annotations (66)

Configuration Parameters (22)
resources.cpu_coresrecommended: 10
minimum CPU cores available for task execution; must match or exceed task requirements
worker-disconnect-delayrecommended: increase above 60s default
Scheduler timeout in [scheduler] section; prevents premature task completion on heartbeat delay
--workersrecommended: 2
CLI parameter for worker count; provides redundancy against single worker disconnection
max_shown_tasksrecommended: 100000
maximum tasks in task_list API to prevent browser freezing
max_graph_nodes
maximum nodes in graph APIs to speed up visualization
worker-keep-aliverecommended: true
prevents workers from exiting when idle with pending tasks
resources.<resource_name>recommended: integer limit based on capacity
Defines maximum concurrent usage of named resource
LUIGI_CONFIG_PATHrecommended: path to config file with [resources] section
Environment variable that both scheduler and workers must reference
--worker-task-limit
Limits tasks per worker to prevent scheduler overload with large task sets
fast_schedulerrecommended: true
Experimental option for O(1) scheduling; disables priorities and resources features
--parallel-scheduling
Does not resolve the O(n) bottleneck per issue report
retry_delayrecommended: 900
Default 900s delay before failed task becomes eligible for retry; set to 0 to disable automatic retry eligibility
--scheduler-retry-delayrecommended: paired with --local-scheduler
controls retry interval for failed tasks; only effective with local scheduler
--local-schedulerrecommended: required when setting custom retry delays via CLI
starts local scheduler instance that respects CLI retry parameters
--scheduler-retry-countrecommended: 10
maximum number of retry attempts for failed tasks
--worker-retry-external-tasks
enables retry behavior for external data dependency tasks
local_schedulerrecommended: False
Use central scheduler to coordinate multiple workers
resourcesrecommended: {'max_workers': 1}
Limit workers to 1 per task to avoid Windows multi-worker issues
scheduler.disable-persist-secondsrecommended: 1200
Duration task remains disabled before automatic retry; increase to allow more manual intervention time
scheduler.retry_countrecommended: 3
Number of retries before task is disabled
scheduler.retry_delayrecommended: 60
Delay in seconds between retry attempts
scheduler.disable-hard-timeoutrecommended: 21600
Hard timeout for disabled state in seconds
Error Signatures (10)
There are no more tasks to run at this timelog pattern
pending tasks possibly being run by other workerslog pattern
pending tasks unique to this workerlog pattern
were left pending because of unknown reasonlog pattern
TypeError: __str__ returned non-string (type NoneType)exception
500 POST /api/get_workhttp status
Uncaught exception POST /api/get_worklog pattern
This progress looks :| because there were tasks that were not granted run permission by the schedulerlog pattern
DEBUG: Pending tasks: 1log pattern
Data for FileTask(filename=test-file.json) does not exist (yet?). The task is an external data dependency, so it cannot be run from this luigi process.log pattern
CLI Commands (6)
luigi --module all_reports RangeDailyBase --of AllReports --start 2015-01-01diagnostic
luigi --module all_reports RangeDaily --of AllReports --start 2015-01-01remediation
--workers 2remediation
luigi --module x examples.X --worker-keep-aliveremediation
python -m luigi --module luigi_external MasterTask --worker-retry-external-tasks --scheduler-retry-delay 1 --scheduler-retry-count 10 --worker-keep-alive --local-schedulerremediation
luigiddiagnostic
Technical References (28)
luigi.Taskcomponent[resources]componentbulk_completecomponent/api/get_workcomponentscheduler.pyfile pathcount_pendingcomponentget_workcomponentWorkercomponentcentral schedulercomponentschedulercomponentPENDINGconceptRUNNINGconceptluigidcomponentluigi.cfgfile pathworker process poolcomponentworker.pyfile pathrequires()componentRange modulecomponentScheduler.get_work()component_upstream_status()componentWrapperTaskcomponentDateParametercomponentexamples/scheduler/query_scheduler.pyfile pathExternalTaskcomponentDAGconceptworkercomponentdisable statusconceptpending statusconcept
Related Insights (21)
Resource requirements block task scheduling despite available workerscritical
▸
Recurring task failures cause missing deliverables for date rangeswarning
▸
Scheduler overload from redundant completeness checks with large task setswarning
▸
Worker disconnection causes parent task to complete prematurely with pending childrenwarning
▸
Visualizer frozen or slow with large task graphsinfo
▸
Resource limits throttle task execution when capacity exceededwarning
▸
Scheduler API TypeError prevents worker task assignmentcritical
▸
Worker disconnect leaves tasks incomplete when jobs finish earlywarning
▸
Scheduler blocks tasks when resource limits are exceededwarning
▸
Resource limits not enforced when scheduler lacks config file accesswarning
▸
Worker process does not scale beyond thousands of taskswarning
▸
Tasks not executed despite met dependenciescritical
▸
Worker-side blocking causes tasks to remain pendingwarning
▸
Scheduler becomes unworkable when requiring 100,000+ taskscritical
▸
Scheduler.get_work() has O(n) complexity causing multi-second delayswarning
▸
Task stuck Pending due to non-deterministic requires() methodwarning
▸
Failed tasks display as Pending during retry windowinfo
▸
Programmatic scheduler status monitoring alternativeinfo
▸
Scheduler retry-delay parameter requires local-scheduler flag to take effectwarning
▸
Multiple workers on Windows cause scheduling and state synchronization failurescritical
▸
Failed tasks retry infinitely after disable suspension expireswarning
▸