Apache DataFusion

Blocking memory allocation causes deadlock risk when waiting for memory

warning
availabilityUpdated Mar 5, 2026(via Exa)
How to detect:

DataFusion currently errors immediately when memory is unavailable. The proposed ability to wait for memory to be freed introduces deadlock potential if multiple operators wait on each other to release memory, particularly in multi-partition scenarios.

Recommended action:

If enabling opt-in memory waiting behavior, implement strict deadlock detection. Monitor datafusion.operator.elapsed_time for operators stuck waiting on memory. Default behavior should remain immediate error on memory exhaustion. Only enable waiting with careful consideration of query complexity and partition count.