Apache DataFusion

DataFusion crashes with assertion failures when batch_size differs from default 8192

critical
configurationUpdated Mar 12, 2026(via Exa)
How to detect:

When DataFusion's batch_size configuration is set to non-default values (e.g., batch_size = 1), the system crashes with assertion failures in arrow-select operations. The failure manifests as 'assertion `left == right` failed' with mismatched expected values (e.g., left: 4, right: 2), causing panics in tokio worker threads.

Recommended action:

1. Verify current batch_size configuration in config.rs. 2. If using non-default batch_size values, revert to default (8192) until Arrow-rs issue #9506 fix is incorporated. 3. Run 'cargo test' to validate stability. 4. Monitor for Arrow-rs version bump that includes the fix from apache/arrow-rs#9506. 5. After Arrow-rs upgrade, re-test with custom batch_size values.