Driver Bottleneck Under High Task Volume

When processing thousands of small files or high-cardinality shuffles, the Spark driver becomes CPU and memory saturated, causing executors to sit idle despite available capacity. Job duration extends 3-4x beyond expected runtime.

Databricks insight details requires a free account. Sign in with Google or GitHub to access the full knowledge base.