Technologies/Luigi/luigi.resources.used
LuigiLuigiMetric

luigi.resources.used

Resources currently in use
Dimensions:None
Available on:PrometheusPrometheus (1)
Interface Metrics (1)
PrometheusPrometheus
Current amount of resource in use across all running tasks
Dimensions:None
Sources
luigi_resources_usedgithub.com

Technical Annotations (24)

Configuration Parameters (6)
resources.cpu_coresrecommended: 10
minimum CPU cores available for task execution; must match or exceed task requirements
[resources]
section in luigi.cfg defining resource limits
resources.<resource_name>recommended: integer limit based on capacity
Defines maximum concurrent usage of named resource
LUIGI_CONFIG_PATHrecommended: path to config file with [resources] section
Environment variable that both scheduler and workers must reference
apirecommended: 1
In [resources] config section; limits available API resource pool
resourcesrecommended: {"api": 1}
Task class attribute; declares how much API resource each task instance consumes
Error Signatures (6)
There are no more tasks to run at this timelog pattern
pending tasks possibly being run by other workerslog pattern
pending tasks unique to this workerlog pattern
RuntimeError: Unfulfilled dependencies at run timeexception
25exit code
This progress looks :| because there were tasks that were not granted run permission by the schedulerlog pattern
Technical References (12)
luigi.Taskcomponent[resources]componentdependenciesconceptcontainercomponentcentral schedulercomponentMesoscomponentresourcescomponentluigi.cfgfile pathschedulercomponentPENDINGconceptRUNNINGconceptluigidcomponent
Related Insights (11)
Resource requirements block task scheduling despite available workerscritical
▸
Luigi tasks fail with unfulfilled dependency errors under resource contentionwarning
▸
Resource exhaustion prevents task executionwarning
▸
Resource limits throttle task execution when capacity exceededwarning
▸
Multiple workers can overload system without global resource limitswarning
▸
Resource limits prevent task parallelism overloadinfo
▸
Unspecified resources default to limit 1, causing unexpected task serializationwarning
▸
Scheduler blocks tasks when resource limits are exceededwarning
▸
Resource limits not enforced when scheduler lacks config file accesswarning
▸
API rate limit exceeded due to uncontrolled task concurrencywarning
▸
Unconfigured resource limits enable task contentionwarning
▸