Trino

Memory limit exceeded after upgrade due to missing Iceberg partition filter pushdown

critical
performanceUpdated Nov 14, 2024
Technologies:
How to detect:

After upgrading from Trino 419 to 464, queries joining Iceberg tables fail with 'Query exceeded distributed user memory limit of 40GB' when using DATE() function on partition columns with dynamic date expressions (e.g., current_date - INTERVAL '60' DAY). Query optimizer fails to push down partition filters to Iceberg connector, causing excessive data scanning (531M rows, 10.7GB vs expected 127M rows, 1.52GB). Explain plan shows missing filter predicates that were present in version 419.

Recommended action:

Rewrite query predicates to use literal date values instead of dynamic expressions with DATE() casting. Change WHERE DATE(o.day_partition) >= current_date - INTERVAL '60' DAY to WHERE o.day_partition >= '2024-09-15' (compute literal date in application). Verify fix by comparing explain plans to confirm partition filter pushdown is occurring. If dynamic dates are required, increase distributed user memory limit as temporary workaround while investigating query optimizer regression.