Trino

Insufficient buffer sizes prevent large dataset operations

warning
Resource ContentionUpdated Nov 17, 2024(via Exa)
Technologies:
How to detect:

Default buffer sizes for exchange (~32MB), sink (~32MB), and deduplication (~32MB) are too small for queries processing large datasets. Operations like sorting on tables >30GB particularly affected. Multiple users report doubling buffer sizes resolves intermittent PAGE_TRANSPORT_TIMEOUT failures.

Recommended action:

On worker nodes, increase: exchange.max-buffer-size to 64MB or 5120MB, sink.max-buffer-size to 64MB or 5120MB, exchange.deduplication-buffer-size to 64MB or 5120MB, driver.max-page-partitioning-buffer-size to 5120MB. For broadcast operations, set sink.max-broadcast-buffer-size to 512MB. Set exchange.max-response-size to 32MB on coordinator. Tune based on typical dataset sizes.