Technologies/Trino/trino.memory.queries_killed_oom
TrinoTrinoMetric

trino.memory.queries_killed_oom

Queries killed OOM
Dimensions:None
Available on:Native (1)DatadogDatadog (1)
Interface Metrics (2)
Native
Total number of queries killed due to out-of-memory conditions
Dimensions:None
DatadogDatadog
Total count of queries killed due to out of memory conditions
Dimensions:None
Sources
trino.memory.ClusterMemoryManager.QueriesKilledDueToOutOfMemorytrino.io
trino.memory.queries_killed_due_to_out_of_memorydocs.datadoghq.com

Technical Annotations (28)

Configuration Parameters (12)
task.low-memory-killer.policyrecommended: total-reservation-on-blocked-nodes
Enables automatic query termination before memory exhaustion with TASK retry policy
retry-policyrecommended: TASK
Enables task-level retries requiring memory management
hive.collect-column-statistics-on-writerecommended: Evaluate disabling if data quality issues persist
Auto-collects column stats that can become skewed with outliers
query.low-memory-killer.policyrecommended: total-reservation-on-blocked-nodes
default policy kills highest-memory query on OOM nodes
query.low-memory-killer.delayrecommended: 5m
Grace period before killing OOM queries
spill_enabledrecommended: true
Enables spilling intermediate results to disk when memory is exhausted
spiller-spill-pathrecommended: /tmp/trino-spill
Directory path for temporary spill files
query.max-memoryrecommended: 50GB
Maximum memory per query to prevent single query from consuming all cluster memory
aggregation_operator_unspill_memory_limitrecommended: 4GB
Memory threshold for aggregation operators before unspilling
query_max_memory_per_noderecommended: 8GB
Maximum memory per node for each query
task.min-writer-countrecommended: 1
default value, reduce if OOM during writes to partitioned tables
task.max-writer-count
default is node CPUs (min 2, max 64), reduce if OOM during writes
Error Signatures (3)
Query exceeded per-node memory limitexception
out of memoryexception
QUERY_CANCELEDerror code
CLI Commands (3)
EXPLAINdiagnostic
SHOW STATSdiagnostic
SELECT query_id, user, peak_memory_bytes / 1024 / 1024 AS peak_memory_mb FROM trino_events.trino_queries WHERE create_time >= now() - interval '7' day ORDER BY peak_memory_mb DESC LIMIT 10;diagnostic
Technical References (10)
broadcast joinconceptcost-based optimizercomponentHive Metastorecomponentcardinality estimationconcepttrino_events.trino_queriescomponentpeak_memory_bytescomponentOOMconceptretry-policyconfigurationpreferred partitioningconcepttask writer scalingconcept
Related Insights (13)
Memory exhaustion requires manual query kills with TASK retry policycritical
▸
Worker resource exhaustion causes complete query failurecritical
▸
TASK retry policy without memory killer requires manual interventioncritical
▸
Skewed column statistics cause broadcast join memory overflowcritical
▸
High memory queries risk OOM and cluster instabilitycritical
▸
Out-of-memory conditions kill running queriescritical
▸
Low memory triggers query killer on out-of-memory nodescritical
▸
Memory exhaustion triggers query and task terminationcritical
▸
Memory exhaustion causes query failures in complex analytical workloadswarning
▸
Out of memory from excessive concurrent writers on partitioned tablescritical
▸
High memory usage causes out-of-memory failurescritical
▸
Queries killed by low memory killer during OOM conditionscritical
▸
Queries canceled due to memory or CPU limit exceededwarning
▸