Multi-partition sorting hits memory bugs causing spill coordination failures
criticalResource ContentionUpdated Mar 5, 2026(via Exa)
Technologies:
How to detect:
When sorting with multiple partitions, DataFusion experiences bugs in spill coordination. The multi-level merge implementation has acknowledged limitations in handling memory pressure across partitions, leading to potential query failures or memory exhaustion.
Recommended action:
Monitor datafusion.operator.spill_count and datafusion.operator.spilled_bytes across partitions. If queries with multi-partition sorts fail with memory errors, consider reducing partition count or implementing partition-aware memory allocation. Track GitHub issue #20715 for the memory coordinator fix.