Technologies/Apache DataFusion/datafusion.operator.output_rows
Apache DataFusionApache DataFusionMetric

datafusion.operator.output_rows

Rows output by operator
Dimensions:None
Available on:PrometheusPrometheus (1)Native (1)OpenTelemetryOpenTelemetry (1)
Interface Metrics (3)
PrometheusPrometheus
Total number of rows processed by a physical plan operator
Dimensions:None
Native
Number of rows output by a physical plan operator
Dimensions:None
OpenTelemetryOpenTelemetry
Total number of rows produced by a physical operator
Dimensions:None
Sources
datafusion_physical_plan_rows_totalgithub.com
output_rowsgithub.com
datafusion.operator.output_rowsgithub.com

Technical Annotations (41)

Configuration Parameters (4)
datafusion.execution.batch_sizerecommended: 8192
Default batch size for buffer-in-memory batches; increase if creating tiny batches
datafusion.execution.coalesce_batchesrecommended: true
Automatically coalesce small batches between operators
/sys/kernel/mm/transparent_hugepage/enabledrecommended: never
Transparent huge pages cause ~18% overhead via page faults in database workloads
enforce_batch_size_in_joinsrecommended: enabled
Restricts maximum output batch size of join operators to batch_size when OOM occurs
Error Signatures (1)
output_rows=0log pattern
CLI Commands (2)
EXPLAIN ANALYZE SELECT * FROM t1 LEFT ANTI JOIN (SELECT * FROM t2 WHERE k <> 1) t2 ON t1.k = t2.kdiagnostic
datafusion-cli -c "select sum(l_extendedprice) / 7.0 as avg_yearly from lineitem, part where p_partkey = l_partkey and p_brand = 'Brand#23' and p_container = 'MED BOX' and l_quantity < (select 0.2 * avg(l_quantity) from lineitem where l_partkey = p_partkey);"diagnostic
Technical References (34)
dynamic partitioningconceptskipped_aggregation_rows metriccomponentCoalesceBatchesStreamcomponentHashJoincomponentMutableArrayDatacomponentarrow_select::concatcomponentBatchSplittercomponentMemoryPoolcomponentMemoryReservationcomponentRecordBatchcomponentNestedLoopJoinExeccomponentrecord batchconceptprobe-sideconceptbuild-sideconceptbatch_sizeconfiguration parameterCoalesceBatchesExeccomponentBatchCoalescercomponentSortMergeJoincomponentHashJoinExeccomponentLEFT ANTI JOINconceptRIGHT ANTI JOINconceptfast-path optimizationconceptnested_loop_join.rsfile pathbatch_transformercomponentColumnStatisticscomponentTable Statisticscomponentjoin cardinality estimationconceptfilter selectivityconceptTPC-Hconceptbuild sideconceptprobe sideconceptstar schemaconceptright deep treeconceptEXPLAIN ANALYZEcomponent
Related Insights (11)
Streaming query throughput degradation after DataFusion 50 upgradewarning
▸
Tiny output batches cause excessive metadata memory consumptionwarning
▸
Partial aggregation inefficiency with high cardinality causes performance degradationwarning
▸
CoalesceBatches spends 17% of join execution time concatenating small filtered batcheswarning
▸
Cascaded joins produce oversized RecordBatches causing OOMcritical
▸
Nested loop join creates excessive memory usage through oversized record batcheswarning
▸
Join operators produce non-uniform batch sizes causing memory and performance issueswarning
▸
Hash join with empty build side reports zero output rows despite producing datawarning
▸
Nested loop join produces massive intermediate result sets consuming memorywarning
▸
Join disasters can occur without proper join reordering and statisticscritical
▸
Suboptimal join order causes 60% query performance degradation on multi-table joinswarning
▸