Regression introduced in DataFusion 52.3.0 removing Dictionary array flattening

warning

configurationUpdated Mar 13, 2026(via Exa)

Sources

Dynamic filter pushdown fails with "Can't compare arrays of different types" when joining VALUES against Dictionary-encoded Parquet columns · Issue #20937 · apache/datafusiongithub.com

Technologies:

Apache DataFusionsubject

How to detect:

DataFusion versions 52.3.0 and later fail queries with Dictionary-encoded Parquet joins when pushdown_filters is enabled. Versions 52.0.0 through 52.2.0 work correctly. The regression was introduced by PR #20505 which removed flatten_dictionary_array() in inlist_builder.rs, preventing conversion of Dictionary arrays to plain Utf8 before building InList.

Recommended action:

Downgrade to DataFusion 52.2.0 or earlier if encountering this issue and unable to disable pushdown_filters. Track fix in PR #20962 for permanent resolution. The original flattening in inlist_builder.rs converted Dictionary arrays to plain types, ensuring type compatibility in comparisons.