Apache DataFusion

Memory pool lacks dual watermark causing premature OOM errors

warning
Resource ContentionUpdated Mar 5, 2026(via Exa)
How to detect:

DataFusion's current memory pool uses a single memory limit threshold, causing queries to error immediately when reaching capacity. Without separate low watermark (start spilling) and high watermark (error out), operators cannot distinguish between recoverable memory pressure and true out-of-memory conditions.

Recommended action:

Track datafusion.memory_pool.used approaching datafusion.memory_pool.limit. When queries fail with memory errors but spilling could have prevented the failure, investigate memory_reservation_bytes configuration. Plan for dual-watermark system described in issue #20715 to distinguish spill threshold from hard limit.