Apache DataFusion

Multi-partition sorting hits memory bugs causing spill coordination failures

critical
Resource ContentionUpdated Mar 5, 2026(via Exa)
How to detect:

When sorting with multiple partitions, DataFusion experiences bugs in spill coordination. The multi-level merge implementation has acknowledged limitations in handling memory pressure across partitions, leading to potential query failures or memory exhaustion.

Recommended action:

Monitor datafusion.operator.spill_count and datafusion.operator.spilled_bytes across partitions. If queries with multi-partition sorts fail with memory errors, consider reducing partition count or implementing partition-aware memory allocation. Track GitHub issue #20715 for the memory coordinator fix.