Databricks insights

Clusters remain running with minimal utilization (cpu_usage_percentage <10%, spark_executor_activetasks = 0) due to high cluster_auto_termination_minutes settings or disabled auto-termination, burning compute costs without productive work.

Model serving endpoint latency (time_ms) exceeding p99 SLA thresholds causes user-facing application slowdowns. Latency violations often correlate with resource constraints (cpu_usage_percentage, mem_usage_percentage, gpu_usage_percentage) or increased request rates.

Excessive JVM garbage collection time (spark_executor_jvmgctime) consumes CPU cycles and pauses task execution, causing tasks to take longer despite adequate cluster resources. GC time >10% of total execution time indicates memory management issues.

Spark executor memory exhaustion during shuffle operations causes data to spill to disk, dramatically slowing down jobs. High spark_executor_diskused during shuffle-heavy stages indicates memory-to-disk spill, which can be 10-100x slower than in-memory processing.

Query performance suffers when warehouse_compilation_time_ms represents >30% of warehouse_query_time_ms, indicating inefficient query patterns, missing optimization, or complex query logic requiring excessive planning time.

Jobs experience extended queue times when all available clusters are at capacity or when cluster creation rate limits are hit. job_queue_time_ms >5 minutes indicates resource availability issues rather than job execution problems.

Jobs configured without timeout or duration threshold settings can run indefinitely when stuck, burning compute costs for hours or days before detection while potentially causing SLA violations.

Jobs run on schedule updating tables that have zero downstream reads, wasting compute resources on data products with no consumers. These represent immediate cost optimization opportunities.

Jobs running on deprecated or old Databricks runtime versions lack security patches, performance improvements, and face EOL deadlines. Legacy runtimes are difficult to track at scale across hundreds of jobs.

Clusters cycle between scale-up and scale-down events when min_workers is too low for baseline workload or max_workers is hit repeatedly, causing job delays and resource waste. 3-5 minute VM provisioning latency amplifies the impact.

Unexpected increases in Databricks Apps DBU consumption can be detected by monitoring system.billing.usage filtered for billing_origin_product='APPS', enabling proactive cost management and resource optimization.

Serverless workloads without cluster_id or warehouse_id attribution make cost allocation impossible, preventing chargeback models and hiding cost spikes. Usage appears in system.billing.usage but cannot be tied to responsible teams or projects.

Databricks costs increase unexpectedly when workloads shift to different SKUs (e.g., STANDARD to PREMIUM compute, serverless adoption) without visibility into which products drive the change. system.billing.usage shows cost increases but root cause product is unclear without drilling into billing_origin_product and product_features.

Databricks system.billing.usage captures only Databricks-metered storage (default storage API operations), missing cloud provider storage costs (S3, ADLS, GCS) that can represent significant spend. Teams underestimate total cost by 20-40% when relying solely on Databricks billing tables.

Streaming pipelines fall behind ingestion rate as backlog (measured in bytes, records, files, or seconds) continuously grows, indicating processing throughput cannot keep pace with incoming data volume.

Jobs fail or run slowly while cluster CPU and memory utilization remain below 60%, indicating resource allocation mismatch, insufficient parallelism, or driver bottlenecks rather than capacity constraints.

Streaming pipelines accumulate backlog when micro-batch processing time exceeds arrival rate, indicated by growing backlog_bytes and backlog_files. Insufficient executor cores or memory causes throughput to fall behind ingestion rate.

Spark shuffle operations (joins, aggregations) saturate network bandwidth when spark_executor_shuffleread and spark_executor_shufflewrite volumes spike, causing executor task delays. Network becomes bottleneck before CPU or memory pressure appears.

Jobs run beyond expected duration limits but teams lack visibility because Duration Warning notifications are not configured, delaying detection of performance degradation or resource bottlenecks.

When multi-task jobs fail, identifying the specific failing task and root cause requires manual navigation through Databricks UI. Task-level metrics (job_setup_time_ms, job_execution_time_ms per task) are not easily aggregated across runs, slowing incident response.

Individual Spark executors run out of heap memory when processing oversized partitions (>500MB), causing task failures and cascading cluster degradation. Partition size imbalance creates memory pressure that ripples across healthy executors.

Query performance degrades when warehouse_cache_hit_ratio drops below 60-70%, forcing repeated scans of underlying data. Cache misses increase warehouse_execution_time_ms and warehouse_query_time_ms, driving up both latency and cost.

Queries use sub-optimal execution plans because Delta statistics or cost-based optimizer statistics are missing or incomplete, causing full table scans instead of partition pruning and inefficient join strategies.

Multiple jobs or queries writing to the same Delta table simultaneously cause transaction conflicts, leading to automatic retries that increase latency and compute costs. High conflict rates indicate scheduling or design issues.

When processing thousands of small files or high-cardinality shuffles, the Spark driver becomes CPU and memory saturated, causing executors to sit idle despite available capacity. Job duration extends 3-4x beyond expected runtime.

Alert-on-failure strategies generate excessive noise from transient failures (dropped connections, spot instance losses) that auto-recover, causing on-call fatigue and masking real incidents. Event hooks only deliver 90% of critical alerts.

DLT expectations (valid_pricing, valid_driver_age) silently drop records that fail quality checks. Without monitoring expect_or_drop metrics, teams remain unaware of data quality degradation until downstream impacts occur.