Technologies/Luigi/luigi.scheduler.active_tasks
LuigiLuigiMetric

luigi.scheduler.active_tasks

Tasks currently scheduled
Dimensions:None
Available on:DatadogDatadog (1)
Interface Metrics (1)
DatadogDatadog
Number of tasks currently being scheduled
Dimensions:None
Sources
luigi.scheduler.active_tasksgithub.com

Technical Annotations (24)

Configuration Parameters (11)
worker-disconnect-delayrecommended: increase above 60s default
Scheduler timeout in [scheduler] section; prevents premature task completion on heartbeat delay
--workersrecommended: 2
CLI parameter for worker count; provides redundancy against single worker disconnection
rpc-connect-timeoutrecommended: 60.0
increase from default 10.0 when scheduler is overloaded
rpc-retry-attemptsrecommended: 10
increase from default 3 to handle transient scheduler unavailability
rpc-retry-waitrecommended: 60
increase from default 30 seconds between retry attempts
rpc_connect_timeoutrecommended: 10.0
seconds before timing out scheduler API calls
rpc_retry_attemptsrecommended: 3
maximum connection retries before giving up
rpc_retry_waitrecommended: 30
seconds between retry attempts
max_shown_tasksrecommended: 100000
maximum tasks in task_list API to prevent browser freezing
max_graph_nodes
maximum nodes in graph APIs to speed up visualization
local_schedulerrecommended: False
Use central scheduler instead of local for production-like behavior
Error Signatures (5)
were left pending because of unknown reasonlog pattern
Failed connecting to remote schedulerlog pattern
ConnectionError: HTTPConnectionPool(host='localhost', port=8082): Max retries exceededexception
[Errno 111] Connection refusederror code
luigi.rpc.RPCError: Errors (3 attempts) when connecting to remote schedulerexception
CLI Commands (3)
--workers 2remediation
--local-schedulerremediation
luigid --background --pidfile <PATH_TO_PIDFILE> --logdir <PATH_TO_LOGDIR> --state-path <PATH_TO_STATEFILE>remediation
Technical References (5)
central schedulercomponent/api/add_workercomponentTaskProcess.runcomponentluigidcomponentlocal schedulercomponent
Related Insights (9)
Worker disconnection causes parent task to complete prematurely with pending childrenwarning
▸
Luigi worker fails to connect to central scheduler causing task cascade failurecritical
▸
RPC connection timeout to central schedulercritical
▸
Visualizer frozen or slow with large task graphsinfo
▸
Worker process does not scale beyond thousands of taskswarning
▸
Scheduler stuck more frequent with failed tasks presentcritical
▸
Remote scheduler UI inaccessible without port forwardinginfo
▸
Local scheduler in production allows duplicate task executionwarning
▸
Local scheduler fails to distinguish repeated task invocationswarning
▸