luigi.scheduler.queue_length

Total tasks in scheduler queue

Dimensions:None

Available on:

Prometheus

luigi_scheduler_task_queue_length

Current number of tasks in the scheduler queue

Dimensions:None

Sources

Technical Annotations (13)

Configuration Parameters (5)

wait_interval

seconds workers wait before requesting another job when none available

wait_jitterrecommended: 5.0

randomization added to wait_interval to prevent simultaneous requests

[resources]recommended: explicit limits for each resource type

Prevents default limit of 1 from serializing parallel tasks

resources.<resource_name>recommended: integer limit based on capacity

Defines maximum concurrent usage of named resource

--worker-task-limit

Limits tasks per worker to prevent scheduler overload with large task sets

CLI Commands (1)

luigi --module all_reports RangeDaily --of AllReports --start 2015-01-01remediation

Technical References (7)

bulk_completecomponent[resources]componentschedulercomponentPENDINGconceptRUNNINGconceptrequires()componentRange modulecomponent

Scheduler overload from redundant completeness checks with large task setswarning

▸

Worker thundering herd on schedulerinfo

▸

Unspecified resources default to limit 1, causing unexpected task serializationwarning

▸

Scheduler blocks tasks when resource limits are exceededwarning

▸

Scheduler becomes unworkable when requiring 100,000+ taskscritical

▸