spark_executor_active_tasks

When processing thousands of small files or high-cardinality shuffles, the Spark driver becomes CPU and memory saturated, causing executors to sit idle despite available capacity. Job duration extends 3-4x beyond expected runtime.

Jobs fail or run slowly while cluster CPU and memory utilization remain below 60%, indicating resource allocation mismatch, insufficient parallelism, or driver bottlenecks rather than capacity constraints.

Databricks autoscaling takes 3-5 minutes to provision new VMs during demand spikes, causing queued tasks and degraded performance while pending tasks exceed available executor cores.