datafusion.memory_pool.limit

Maximum memory limit

Dimensions:None

Technical Annotations (32)

Configuration Parameters (6)

RuntimeConfig.memory_poolrecommended: FairSpillPool with 2-3x expected memory

Provides headroom for sort_batch memory spike during spill

MemoryConsumer.with_can_spillrecommended: true

Enables spillable consumer for GroupedHashAggregateStream

datafusion.optimizer.prefer_hash_joinrecommended: false

Set to false for memory-constrained workloads to use SortMergeJoin instead

datafusion.memory_pool.limit

Set explicit limit to fail fast rather than exhaust system memory

prefer_hash_join

may need to disable hash joins to avoid memory exhaustion

hash_join_single_partition_threshold

adjust to reduce memory footprint of hash joins

Error Signatures (3)

ResourcesExhausted("Additional allocation failedexception

Failed to allocate additionallog pattern

ResourcesExhaustedexception

Technical References (23)

memory_reservation_bytescomponentRepartitionExeccomponentExternalSortercomponentFairSpillPoolcomponenttry_growcomponentPrometheuscomponentDataDogcomponentGrafanacomponentHashAggregationExeccomponentGroupedHashAggregateStreamcomponentsort_batchcomponentBatchSplittercomponentmemory reservationsconceptHashJoincomponentSortMergeJoincomponentmemory poolcomponentmemory_poolcomponentspillingconceptunnestcomponentGROUP BYcomponentarray_aggcomponentstreaming executionconceptmemory spillingconcept

Related Insights (9)

Memory pool lacks dual watermark causing premature OOM errorswarning

▸

ExternalSort fails when non-spillable input operators exhaust memory poolcritical

▸

Resource exhaustion undetected without metric trackingwarning

▸

Hash aggregation spill doubles memory usage due to sort_batch copycritical

▸

BatchSplitter does not prevent memory issues from oversized join batchescritical

▸

Hash join fails when memory limit exceeded without spilling supportcritical

▸

Memory exhaustion from insufficient memory pool limits on resource-constrained systemscritical

▸

Unnest with GROUP BY causes unbounded memory growth despite streamingcritical

▸

Memory pressure during joins triggers disk spilling with performance impactwarning

▸