Technologies/Apache DataFusion/datafusion.operator.spilled_bytes
Apache DataFusionApache DataFusionMetric

datafusion.operator.spilled_bytes

Bytes spilled to disk
Dimensions:None
Available on:PrometheusPrometheus (1)Native (1)OpenTelemetryOpenTelemetry (1)
Interface Metrics (3)
PrometheusPrometheus
Total bytes spilled to disk by physical plan operators
Dimensions:None
Native
Total bytes written to disk during spill operations
Dimensions:None
OpenTelemetryOpenTelemetry
Total bytes written to disk during spill operations
Dimensions:None
Sources
datafusion_physical_plan_spilled_bytes_totalgithub.com
spilled_bytesgithub.com
datafusion.operator.spilled_bytesgithub.com

Technical Annotations (17)

Configuration Parameters (6)
datafusion.execution.max_spill_file_size_bytesrecommended: 134217728
Default 128 MB balances syscall overhead vs disk space retention
with_disk_manager_osrecommended: enabled
Enables OS-managed temporary disk storage for query spilling
with_fair_spill_poolrecommended: 100000000
Sets spill pool size in bytes (100MB example)
prefer_hash_join
may need to disable hash joins to avoid memory exhaustion
hash_join_single_partition_threshold
adjust to reduce memory footprint of hash joins
collect_left_threshold
Memory threshold determining when to spill build side during dynamic reordering
CLI Commands (1)
writer.write(&batch)diagnostic
Technical References (10)
multi_level_merge.rsfile pathnum_spillcomponentRuntimeConfigcomponentIPCWritercomponentemitcomponenthybrid hash joinconceptspill thresholdconceptmemory_poolcomponentspillingconceptmemory spillingconcept
Related Insights (8)
Multi-partition sorting hits memory bugs causing spill coordination failurescritical
▸
Excessive spill files created when max_spill_file_size_bytes is too lowwarning
▸
Missing disk spill manager prevents large query executionwarning
▸
Large single-batch spill files cause merge failureswarning
▸
Hybrid hash join spill threshold misconfiguration causes premature spillingwarning
▸
Memory exhaustion from insufficient memory pool limits on resource-constrained systemscritical
▸
Memory pressure during joins triggers disk spilling with performance impactwarning
▸
Peak memory consumption during hash joins with poor cardinality estimateswarning
▸