Presto

Query Memory Exhaustion and Distributed Join Imbalance

critical
Resource ContentionUpdated Sep 26, 2024

Presto queries fail with 'Query exceeded max memory size' or 'Query exceeded local memory limit' errors, often caused by inefficient join ordering where larger tables are on the right side, forcing expensive hash joins instead of broadcast joins.

How to detect:

Monitor failed queries with memory limit errors. Check execution traces for high presto_execution_failed_queries_one_minute_rate, presto_memory_reserved_size approaching presto_memory_max_size, and join plans where the right side has significantly higher cardinality than the left side. Look for executor_blocked_splits and executor_waiting_splits increasing.

Recommended action:

Rewrite queries to place smaller tables on the right side of joins to enable broadcast joins. Increase query.max-memory-per-node to 40% of worker instance memory. Use distributed (reduce-side) joins instead of map-side joins for large-large table joins. Consider increasing worker instance size or cluster size.