datafusion.parquet.pushdown_rows_filtered

Rows filtered by pushdown

Dimensions:None

Available on:Native (1)

Interface Metrics (1)

Native

pushdown_rows_filtered

Number of rows filtered by pushdown predicates at the file format level

Dimensions:None

Sources

pushdown_rows_filteredgithub.com

Technical Annotations (22)

Configuration Parameters (6)

datafusion.execution.parquet.pushdown_filtersrecommended: false

default to avoid regressions until performance issues resolved

datafusion.execution.target_partitionsrecommended: 1

set to 1 during benchmarking to reduce variability

datafusion.execution.parquet.binary_as_stringrecommended: true

required for ClickBench data processing

datafusion.optimizer.enable_aggregate_dynamic_filter_pushdownrecommended: false

disable to avoid aggregate filter overhead in Parquet reader

datafusion.execution.parquet.reorder_filtersrecommended: false

Default false; enable to optimize filter ordering when pushdown enabled

with_parquet_pruningrecommended: True

Enables row group and page-level pruning for Parquet files

Error Signatures (1)

lit(true)log pattern

CLI Commands (1)

set datafusion.optimizer.enable_aggregate_dynamic_filter_pushdown = false;diagnostic

Technical References (14)

ArrowPredicate APIcomponentlate materializationconceptRowFiltercomponentdatafusion/physical-plan/src/aggregates/no_grouping.rsfile pathdatafusion/pruning/src/file_pruner.rsfile patharrow-rscomponentpage index pruningconceptfilter_pushdown.rsfile pathphysical-optimizer/filter_pushdown.rsfile pathFileOpener::opencomponentpartition boundsconceptdynamic filterconceptrow group filteringconceptpredicate pushdownconcept

Related Insights (7)

Parquet filter pushdown causes query slowdowns for specific query patternswarning

▸

Aggregate dynamic filters cause overhead when pushed to Parquet readerwarning

▸

Low-selectivity filters cause unnecessary row-level evaluation overheadinfo

▸

Parquet predicate evaluation errors lost when pushdown filters enabledinfo

▸

Disabled Parquet filter pushdown reduces query performancewarning

▸

Dynamic filters not generated or built on wrong table due to lost statisticswarning

▸

Race condition in HashJoin dynamic filter construction prevents immediate probe-side filter visibilitywarning

▸