CockroachDB insights

Open file descriptors approaching the soft limit (sys_fd_open nearing sys_fd_softlimit) indicates impending node failure. When exhausted, the node cannot open new connections, files, or sockets, causing cascading failures.

Mean clock offset (clock_offset_meannanos) exceeding thresholds threatens CockroachDB's correctness guarantees. Clock skew can cause transaction retries, uncertainty intervals, and in extreme cases, consistency violations in distributed transactions.

When admission.io.overload exceeds 1, L0 compactions cannot keep up with write load. Correlate with storage.l0-num-files, storage.l0-sublevels, and rocksdb.read-amplification to assess LSM health. Performance issues escalate to stability issues if sustained.

When sys.runnable.goroutines.per.cpu exceeds 30, CPU is overloaded. Short spikes cause response time inconsistency; sustained overload (tens of seconds to minutes) threatens cluster stability.

Storage I/O Queue Saturation Cascade

Disk queue depth (iopsinprogress) above 5 sustained indicates storage cannot keep up with demand. When combined with low actual IOPS/MBPS, points to provisioning issues or CPU bottleneck preventing I/O generation.

CockroachDB Range Under-Replication Risk

Under-replicated ranges indicate data not conforming to resilience goals. Combined with critical node detection, this reveals single points of failure that could cause data unavailability.

Write-ahead log fsync latency exceeding 100ms indicates disk stall, severely impacting transaction commit latency. This is critical for durability as WAL ensures crash recovery. Without WAL failover configured, single storage device stalls block all writes on the affected node.

When open file descriptors approach the limit, the cluster cannot accept new connections or open additional files. This causes cascading failures as operations requiring file handles fail. Multi-node single-machine testing amplifies this issue as actual FD count is multiplied by node count in metrics.

CockroachDB storage volumes should not exceed 60% utilization (40% free space required). Beyond this threshold, performance degrades and cluster stability risks increase. Unlike memory pressure which throttles gradually, disk exhaustion can cause abrupt node failures.

Range splits occur when data or query load exceeds thresholds, with split heuristic based on QPS. Sudden spikes in range.splits metric indicate resource hotspots forming on specific tables/indexes. While splits are normal optimization, rapid splitting signals potential query pattern issues requiring load redistribution.

When sys.runnable.goroutines.per.cpu exceeds 30, CPU is overloaded with work waiting to execute. Short spikes cause inconsistent response times; sustained overload (tens of seconds to minutes) leads to cluster stability issues. Often correlates with sys.cpu.combined.percent-normalized near 1.0 (100%).

Ranges with fewer live replicas than replication target indicate data not meeting resilience goals. Unlike unavailable ranges (which block queries), under-replicated ranges silently increase vulnerability—one more node failure causes data unavailability and potential loss if quorum cannot be reached.

Elevated rocksdb.compactions count combined with unhealthy LSM metrics (read-amp >20, L0-files >1000, admission.io.overload >1) indicates compactions cannot keep pace with workload, leading to performance degradation and eventual stability issues.

Average runnable goroutines per CPU exceeding 30 indicates CPU overload, causing inconsistent response times in short bursts and potential cluster stability issues if sustained for extended periods.

Sustained high IO queue depth (>5 for network storage, >0 for NVMe) combined with low IOPS/MBPS indicates storage provisioning issues rather than workload spikes, requiring infrastructure changes rather than application tuning.

When disk I/O queue depth exceeds 5-10 and LSM health degrades (L0 files >1000 or read-amp >20), admission control begins throttling requests, creating cascading latency spikes. This condition is often invisible until client timeouts start occurring.

Persistently low sys.rss (RSS memory usage) relative to available RAM indicates underutilized memory that could improve performance via larger block cache or SQL memory pool. Poor block cache hit rates (rocksdb.block.cache.misses) coupled with low memory usage suggests cache is undersized for workload demands.

Rising sql_failure_count (SQL statements returning planning or runtime errors) indicates application-level problems or schema issues affecting user queries. Unlike infrastructure failures, this points to query problems requiring application or database schema investigation.

Active SQL connections exceed 4x the number of vCPUs, leading to CPU saturation, query latency degradation, and eventual LSM health deterioration. This is a primary indicator of CPU resource contention.

Storage engine compaction cannot keep pace with writes, causing LSM inversion indicated by declining block cache hit rate and growing data size. This creates cascading latency issues and can lead to node instability.

High transaction contention (>100ms) causes blocking, increased retries, and degraded user experience. Often caused by long-running transactions competing for the same rows.

Ranges fall below target replication factor, creating availability risk. Often caused by node failures, decommissioning, or insufficient cluster capacity during rebalancing.

DistSQL queries touching multiple nodes experience elevated latency, often due to network issues, cross-region data access, or skewed data distribution causing unbalanced execution.

Sudden increase in crdb_sql_statement_errors indicates application bugs, schema issues, or permission problems causing query failures that degrade user experience.

Disk capacity nearing limits prevents compaction, blocks writes, and can trigger node crashes. Early detection critical to prevent cascading cluster failure. CockroachDB recommends maintaining 40% free space.