Apache DataFusion insights

Open SourceVersions: []32 metrics
Apache DataFusion
Native DataFusion scan performance optimization opportunities identifiedinfo

datafusion.parquet.time_scanningdatafusion.parquet.bytes_scanneddatafusion.operator.elapsed_time+1 moregithub.com
11d ago▸
Apache DataFusion
Dynamic filter pushdown causes type comparison error when joining VALUES against Dictionary-encoded Parquet columnscritical

datafusion.parquet.predicate_evaluation_errorsgithub.com
18d ago▸
Apache DataFusion
Regression introduced in DataFusion 52.3.0 removing Dictionary array flatteningwarning

github.com
18d ago▸
Apache DataFusion
DataFusion crashes with assertion failures when batch_size differs from default 8192critical

datafusion.operator.input_batchesgithub.com
19d ago▸
Apache DataFusion
Explain plan validation tests fail with non-default batch_sizewarning

github.com
19d ago▸
Apache DataFusion
GROUP BY ALL with aliased aggregates causes physical plan errorwarning

github.com
19d ago▸
Apache DataFusion
Hash join optimizer selects non-partitioned mode causing 52x slower query executioncritical

datafusion.join.build_input_rowsdatafusion.query.execution_timedatafusion.join.total_timegithub.com
21d ago▸
Apache DataFusion
Aggregation operations under-partition causing multi-fold performance degradationwarning

datafusion.aggregate.groupsdatafusion.query.execution_timegithub.com
21d ago▸
Apache DataFusion
concat_batches overhead causes 33x query performance degradationcritical

datafusion.query.execution_timedatafusion.operator.input_batchesgithub.com
21d ago▸
Apache DataFusion
Hash join with empty build side reports zero output rows despite producing datawarning

datafusion.operator.output_rowsdatafusion.join.build_input_rowsgithub.com
23d ago▸
Apache DataFusion
Unnest with GROUP BY causes unbounded memory growth despite streamingcritical

datafusion.memory_pool.useddatafusion.operator.memory_useddatafusion.memory_pool.limit+1 moregithub.com
24d ago▸
Apache DataFusion
Multi-partition sorting hits memory bugs causing spill coordination failurescritical

datafusion.operator.spill_countdatafusion.operator.spilled_bytesdatafusion.memory_pool.usedgithub.com
26d ago▸
Apache DataFusion
Memory pool lacks dual watermark causing premature OOM errorswarning

datafusion.memory_pool.useddatafusion.memory_pool.limitdatafusion.memory_pool.reservedgithub.com
26d ago▸
Apache DataFusion
Fair pool unfairly allocates memory between spillable and non-spillable operatorswarning

datafusion.operator.memory_useddatafusion.operator.spill_countgithub.com
26d ago▸
Apache DataFusion
Blocking memory allocation causes deadlock risk when waiting for memorywarning

datafusion.operator.elapsed_timedatafusion.memory_pool.usedgithub.com
26d ago▸
Apache DataFusion
Spillable aggregation produces duplicate group keys due to internal state mismatchcritical

datafusion.operator.spill_countdatafusion.aggregate.groupsgithub.com
26d ago▸
Apache DataFusion
Hash collisions can trigger aggregation correctness issues during testinginfo

github.com
26d ago▸
Apache DataFusion
TopK operator panics on Utf8 string column overflow beyond i32::MAXcritical

datafusion.operator.memory_usedgithub.com
1mo ago▸
Apache DataFusion
Pipeline run failure detection via Cloud Monitoring metricscritical

oneuptime.com
1mo ago▸
Apache DataFusion
Pipeline duration exceeds baseline indicating performance degradationwarning

oneuptime.com
1mo ago▸
Apache DataFusion
Record count drop-off between pipeline stages indicates data losswarning

oneuptime.com
1mo ago▸
Apache DataFusion
Error record count exceeds threshold indicating data quality issueswarning

oneuptime.com
1mo ago▸
Apache DataFusion
Connection timeout errors prevent pipeline from reaching data sourcescritical

oneuptime.com
1mo ago▸
Apache DataFusion
Authentication failure blocks database accesscritical

oneuptime.com
1mo ago▸
Apache DataFusion
Schema mismatch causes transformation failureswarning

oneuptime.com
1mo ago▸
Apache DataFusion
Source stage failures from connectivity or query errorscritical

oneuptime.com
1mo ago▸
Apache DataFusion
BigQuery sink failures from permission or schema conflictscritical

oneuptime.com
1mo ago▸
Apache DataFusion
Out of memory errors terminate pipeline executioncritical

oneuptime.com
1mo ago▸
Apache DataFusion
Parquet filter pushdown causes query slowdowns for specific query patternswarning

datafusion.parquet.pushdown_eval_timedatafusion.parquet.pushdown_rows_filtereddatafusion.query.execution_timegithub.com
1mo ago▸
Apache DataFusion
Aggregate dynamic filters cause overhead when pushed to Parquet readerwarning

datafusion.parquet.pushdown_rows_filteredgithub.com
1mo ago▸