concat_batches overhead causes 33x query performance degradation

critical

performanceUpdated Mar 10, 2026(via Exa)

Sources

Dynamic switching of non-partitioned to partitioned for join and aggregation · Issue #20847 · apache/datafusiongithub.com

Technologies:

Apache DataFusionsubject

How to detect:

Query 99 performance overhead mostly caused by concat_batches operation on all values. Execution time improves from 10774ms to 323ms (33.35x faster) when avoiding this concatenation pattern.

Recommended action:

Review query plans for excessive batch concatenation operations. Monitor datafusion.operator.input_batches and datafusion.operator.output_rows metrics. Consider PR #20944 fixes for avoiding concat_batches on value arrays. Profile queries showing high overhead in batch processing operations.