Technologies/Apache DataFusion/datafusion.operator.spill_count
Apache DataFusionApache DataFusionMetric

datafusion.operator.spill_count

Number of spill events
Dimensions:None
Available on:PrometheusPrometheus (1)Native (1)OpenTelemetryOpenTelemetry (1)
Interface Metrics (3)
PrometheusPrometheus
Total number of times data was spilled to disk
Dimensions:None
Native
Number of times data was spilled to disk due to memory pressure
Dimensions:None
OpenTelemetryOpenTelemetry
Number of times this operator spilled data to disk
Dimensions:None
Sources
datafusion_physical_plan_spill_count_totalgithub.com
spill_countgithub.com
datafusion.operator.spill_countgithub.com

Technical Annotations (50)

Configuration Parameters (16)
memory_pool.soft_limitrecommended: for spillable operators only
proposed soft limit allows non-spillable to exceed without error
memory_pool.hard_limitrecommended: soft_limit * 120%
proposed hard limit for total memory including non-spillable
datafusion.execution.max_spill_file_size_bytesrecommended: 134217728
Default 128 MB balances syscall overhead vs disk space retention
datafusion.execution.sort_spill_reservation_bytesrecommended: 10485760
Default 10 MB reserved for in-memory merge during sort spilling
with_disk_manager_osrecommended: enabled
Enables OS-managed temporary disk storage for query spilling
with_fair_spill_poolrecommended: 100000000
Sets spill pool size in bytes (100MB example)
batch_sizerecommended: increase from small values (e.g., 2)
Small batch sizes combined with low memory limits trigger spilling more frequently
memory_pool.limitrecommended: increase from 1600 bytes minimum
Memory pool size of 1600 bytes is insufficient and triggers premature spilling in GroupedHashAggregateStream
RuntimeConfig.memory_poolrecommended: FairSpillPool with 2-3x expected memory
Provides headroom for sort_batch memory spike during spill
MemoryConsumer.with_can_spillrecommended: true
Enables spillable consumer for GroupedHashAggregateStream
sort_spill_reservation_bytes
Controls memory reservation for spill operations during joins
memory_limit
Sets upper bound on memory pool to trigger spilling
prefer_hash_join
controls default join algorithm selection between hash and sort-merge
hash_join_single_partition_threshold
threshold for switching to single-partition hash join
hash_join_single_partition_threshold_rows
row count threshold for single-partition hash join selection
collect_left_threshold
Memory threshold determining when to spill build side during dynamic reordering
Error Signatures (5)
ResourcesExhausted("Additional allocation failedexception
Failed to allocate additionallog pattern
ArrowError(InvalidArgumentError("number of columns(3) must match number of fields(2) in schema"), None)exception
number of columns must match number of fields in schemalog pattern
ResourcesExhaustedexception
Technical References (29)
multi_level_merge.rsfile pathnum_spillcomponentfair poolcomponentspillable consumerconceptRepartitionExeccomponentExternalSortercomponentFairSpillPoolcomponenttry_growcomponentMemoryPoolcomponentRuntimeConfigcomponentGroupValuesColumncomponentvectorized_interncomponentGroupOrdering::FullcomponentGroupedHashAggregateStreamcomponentAggregateExeccomponentdatafusion/physical-plan/src/aggregates/row_hash.rsfile pathHashAggregationExeccomponentsort_batchcomponentgroup_valuescomponentspillingconceptmemory_poolcomponenthybrid hash joinconceptspill thresholdconceptMemoryReservationcomponentpartitioned hash joinconceptJoinPlanner traitcomponenthash joincomponentsort-merge joincomponentmemory spillingconcept
Related Insights (18)
Multi-partition sorting hits memory bugs causing spill coordination failurescritical
▸
Fair pool unfairly allocates memory between spillable and non-spillable operatorswarning
▸
ExternalSort fails when non-spillable input operators exhaust memory poolcritical
▸
FairSpillPool allows premature OOM failures on non-spillable operatorswarning
▸
Excessive spill files created when max_spill_file_size_bytes is too lowwarning
▸
Sort operations run out of memory when sort_spill_reservation_bytes is insufficientcritical
▸
Missing disk spill manager prevents large query executionwarning
▸
Spillable aggregation produces duplicate group keys due to internal state mismatchcritical
▸
Schema mismatch causes GroupedHashAggregateStream spill failure with multiple aggregationscritical
▸
Hash aggregation spill doubles memory usage due to sort_batch copycritical
▸
Batch size checks prevent small spills under memory pressurewarning
▸
Out of memory on large table joins with 16GB RAMcritical
▸
Hybrid hash join spill threshold misconfiguration causes premature spillingwarning
▸
Partitioned hash join memory coordination failure with shared poolwarning
▸
Memory exhaustion from insufficient memory pool limits on resource-constrained systemscritical
▸
Join algorithm selection impacts memory usage vs CPU performance tradeoffswarning
▸
Memory pressure during joins triggers disk spilling with performance impactwarning
▸
Peak memory consumption during hash joins with poor cardinality estimateswarning
▸