Apache DataFusion

Fair pool unfairly allocates memory between spillable and non-spillable operators

warning
Resource ContentionUpdated Mar 5, 2026(via Exa)
How to detect:

The fair memory pool implementation cannot properly distribute memory when operators have mixed spillable and non-spillable portions. Consumers must register as either spillable or non-spillable, but each has both types of memory, leading to unfair allocation and potential starvation.

Recommended action:

Monitor memory distribution across operators using datafusion.operator.memory_used. If some operators consume disproportionate memory while others spill excessively, the fair pool allocation is skewed. Check whether operators are registered correctly as spillable vs non-spillable consumers.