Trino

Worker parallelism drops to zero with threads stuck in WAITING state

critical
performanceUpdated Apr 21, 2025
Technologies:
How to detect:

Worker parallelism drops to 0 and query splits stop executing. Nearly all worker threads become stuck in WAITING and TIMED_WAITING states, preventing query execution despite available cluster resources (CPU < 30%, memory < 40% utilized).

Recommended action:

Collect thread dumps from worker nodes to identify blocking operations or deadlocks. Review task.concurrency and task.max-drivers-per-task settings. Check for connection pool exhaustion, network issues between coordinator and workers, or external dependency timeouts (e.g., metastore). Monitor task.http-timeout-threads and task.http-response-threads for saturation.