Technologies/Luigi/luigi.task.failed.total
LuigiLuigiMetric

luigi.task.failed.total

Total tasks failed
Dimensions:None
Available on:PrometheusPrometheus (1)DatadogDatadog (1)
Interface Metrics (2)
PrometheusPrometheus
Total number of tasks that failed since scheduler startup
Dimensions:None
DatadogDatadog
Number of tasks that have failed
Dimensions:None
Sources
luigi_task_failed_totalgithub.com
luigi.scheduler.failed_tasksgithub.com

Technical Annotations (57)

Configuration Parameters (21)
WORKSPACE
environment variable that namespaces output tables (e.g., 'production', 'peach', ticket number)
SOURCE_ENVIRONMENT
determines which data source schemas to query (staging vs production databases)
worker_disconnect_delayrecommended: 60
seconds to wait before marking disconnected worker's tasks as failed
retry_count
number of failures within disable_window before auto-disable (unset = no auto-disable)
disable_windowrecommended: 3600
time window in seconds for counting failures toward auto-disable
disable_hard_timeout
seconds after first failure before any subsequent failure triggers disable
disable_persistrecommended: 86400
how long auto-disabled tasks remain disabled (1 day default)
retry_external_tasksrecommended: false
whether to continuously recheck incomplete external tasks during execution
keep_aliverecommended: false
must be true for retry_external_tasks to work effectively
check_complete_on_runrecommended: false
whether to verify outputs exist after task completes
check_unfulfilled_depsrecommended: true
whether to verify dependencies are complete before running task
timeoutrecommended: 0
global default task timeout in seconds (0 = no timeout)
disable-hard-timeout
time threshold before permanent task disabling
retry_delayrecommended: 900
Default 900s delay before failed task becomes eligible for retry; set to 0 to disable automatic retry eligibility
workersrecommended: 1 or omit on Windows
Multiple workers not supported on Windows due to lack of process forking
scheduler.retry_countrecommended: 3
number of retry attempts for failed tasks
scheduler.retry-delayrecommended: 1
delay in seconds between retry attempts
keep-aliverecommended: True
Maintains task status to enable retry functionality
scheduler.disable-persist-secondsrecommended: 1200
Duration task remains disabled before automatic retry; increase to allow more manual intervention time
scheduler.retry_delayrecommended: 60
Delay in seconds between retry attempts
scheduler.disable-hard-timeoutrecommended: 21600
Hard timeout for disabled state in seconds
Error Signatures (6)
Luigi Framework Errorexception
RuntimeError: Unfulfilled dependencies at run timeexception
20exit code
BrokenPipeErrorexception
This progress looks :( because there were failed taskslog pattern
status FAILEDlog pattern
CLI Commands (3)
luigi --module all_reports RangeDailyBase --of AllReports --start 2015-01-01diagnostic
luigi --module my_module MyTask --workers Ndiagnostic
pip install -U luigiremediation
Technical References (27)
task schedulercomponenttask signatureconceptluigi.Event.FAILUREcomponentevent_handlercomponentWrapperTaskcomponentrequires()componenttrigger_eventcomponentdependenciesconceptcontainercomponentquery_mapping_datacomponentIAM rolescomponentOIDCprotocolcentral schedulercomponent@luigi.Task.event_handlercomponentWorkercomponentSimpleTaskState.fail_dead_worker_taskcomponentRetryPolicycomponentTaskcomponentTaskProcess.runcomponentscheduler.pyfile pathforkingconceptmultiprocessingcomponentschedulercomponentretry mechanismconcepttask hierarchyconceptdisable statusconceptpending statusconcept
Related Insights (26)
Failed Luigi workflows can be resumed by re-running with same parametersinfo
▸
Failure event handlers not triggered with multiple workerswarning
▸
WrapperTask failures in requires() method do not trigger FAILURE eventswarning
▸
Luigi tasks fail with unfulfilled dependency errors under resource contentionwarning
▸
Undefined query variables halt pipeline executioncritical
▸
Task duration and failure monitoring critical for data pipeline healthwarning
▸
Recurring task failures cause missing deliverables for date rangeswarning
▸
Expired or shared credentials cause Luigi pipeline failurescritical
▸
Worker disconnect marks all running tasks as failedcritical
▸
Automatic task disable after repeated failurescritical
▸
Incomplete external tasks block workflow progresswarning
▸
Task outputs missing after successful runwarning
▸
Unfulfilled dependencies detected at task runtimewarning
▸
Long-running tasks killed by worker timeoutwarning
▸
Task permanently disabled after repeated failures exceed timeoutcritical
▸
Task failure without event handlers prevents proper resource cleanupwarning
▸
Single-threaded task saturates API rate limitswarning
▸
Worker disconnection leaves tasks orphanedwarning
▸
Task retry exhaustion from excessive failureswarning
▸
Scheduler stuck more frequent with failed tasks presentcritical
▸
Failed tasks display as Pending during retry windowinfo
▸
Multiple worker processes fail on Windows with BrokenPipeErrorcritical
▸
Luigi reports failure when retried tasks succeedwarning
▸
Central scheduler fails to update task status after successful retrywarning
▸
Prerequisite task failures block all downstream taskswarning
▸
Failed tasks retry infinitely after disable suspension expireswarning
▸