Technologies/Trino/trino.jvm.heap_memory.used
TrinoTrinoMetric

trino.jvm.heap_memory.used

Heap memory used
Dimensions:None
Available on:Native (1)
Interface Metrics (1)
Native
Amount of heap memory currently used by the JVM
Dimensions:None
Sources
java.lang.Memory.HeapMemoryUsage.usedtrino.io

Technical Annotations (49)

Configuration Parameters (23)
task.low-memory-killer.policyrecommended: total-reservation-on-blocked-nodes
Enables automatic query killing when using TASK retry policy to prevent manual intervention
retry-policyrecommended: TASK
TASK retry policy requires proper memory killer configuration
http.client.timeoutrecommended: 10000ms
default timeout before PAGE_TRANSPORT_TIMEOUT occurs
query.remote-task.max-error-durationrecommended: 1m
Maximum time to wait before abandoning a remote task, may cause premature failures
query-retry-attemptsrecommended: 2
Number of retry attempts, could contribute to retry loops if failures are consistent
-Xmx110Grecommended: 110G
leaves only 15G for OS and off-heap on 125G machine
-XX:G1HeapRegionSizerecommended: 32M
G1 region size for 110G heap; tuning may affect GC performance
-XX:InitialRAMPercentagerecommended: 80
initial heap as percentage of RAM
-XX:MaxRAMPercentagerecommended: 80
max heap as percentage of RAM
memory.heap-headroom-per-noderecommended: increase if pod OOM-killed
reserves native memory outside JVM heap to prevent k8s OOMKills
-Xmxrecommended: 105G
JVM max heap size; must accommodate query.max-memory-per-node + memory.heap-headroom-per-node with buffer for GC
query.max-memory-per-noderecommended: 85GB
Maximum query memory per node; tight allocation relative to heap can trigger excessive GC
query.remote-task.enable-adaptive-request-sizerecommended: true
enables splitting requests to prevent OOM with large schemas
query.remote-task.max-request-sizerecommended: 8MB
maximum size of a single remote task request
query.remote-task.guaranteed-splits-per-taskrecommended: 3
minimum splits per task to ensure workload distribution
query.remote-task.request-size-headroomrecommended: 2MB
headroom beyond request data size
query.low-memory-killer.policyrecommended: total-reservation-on-blocked-nodes
default policy kills highest-memory query on OOM nodes
query.low-memory-killer.delayrecommended: 5m
Grace period before killing OOM queries
spill_enabledrecommended: true
Enables spilling intermediate results to disk when memory is exhausted
spiller-spill-pathrecommended: /tmp/trino-spill
Directory path for temporary spill files
query.max-memoryrecommended: 50GB
Maximum memory per query to prevent single query from consuming all cluster memory
aggregation_operator_unspill_memory_limitrecommended: 4GB
Memory threshold for aggregation operators before unspilling
query_max_memory_per_noderecommended: 8GB
Maximum memory per node for each query
Error Signatures (8)
PAGE_TRANSPORT_TIMEOUTerror code
Failing abandoned tasklog pattern
Total timeout 10000 ms elapsedexception
OOMerror code
Expected response code from http://.*:8080/v1/task/.*/status to be 200, but was 408http status
Error 408 Timeout: Timed outhttp status
io.trino.spi.TrinoExceptionexception
QUERY_CANCELEDerror code
Technical References (18)
PLANNEDcomponentStage 1componentHttpPageBufferClientcomponentJVMcomponentMetabase v0.51.12componentStarburst Metabase driver v5.0componentTrino v457componentG1 garbage collectorcomponentMajor GCconceptOOMKilledconceptcom.oracle.truffle.js.runtime.builtins.JSFunctionObject$Unboundcomponentcom.oracle.truffle.api.strings.TruffleStringcomponentGraalVMcomponentRangercomponentG1 Old Generationcomponent/v1/task/.../statusfile pathleast-waste policyconceptretry-policyconfiguration
Related Insights (16)
TASK retry policy without memory killer requires manual interventioncritical
▸
Worker stuck in PLANNED state after crash recoverycritical
▸
JVM memory allocation exceeds safe threshold causing OOM riskcritical
▸
Cluster intermittently enters idle state after initial query executioncritical
▸
High memory utilization concurrent with CPU saturationwarning
▸
Worker OOM causes pod restarts and query failurescritical
▸
Coordinator crashes from Ranger plugin JavaScript policy memory leakcritical
▸
Long GC pauses cause HTTP 408 task timeout failurescritical
▸
Out-of-memory errors with large schemas in remote taskscritical
▸
Low memory triggers query killer on out-of-memory nodescritical
▸
Low memory triggers task killer on out-of-memory nodescritical
▸
Memory exhaustion triggers query and task terminationcritical
▸
Large schemas cause out-of-memory errors without adaptive request sizingwarning
▸
Memory exhaustion causes query failures in complex analytical workloadswarning
▸
Out-of-memory errors with large schemas when adaptive requests disabledwarning
▸
Queries canceled due to memory or CPU limit exceededwarning
▸