Cassandra insights

Trace Completion Rate Degradationcritical

When the ratio of saved traces to received spans drops below 95%, traces are being lost before storage, indicating storage backend issues, network timeouts, or collector resource constraints.

Storage Backend Write Latency Bottleneckcritical

Slow storage write operations block collector workers, causing span reception to slow and queues to back up, ultimately leading to dropped traces.

Query Service Latency Degradationwarning

High P95/P99 query latencies (>5 seconds) make Jaeger UI unusable during incident troubleshooting, typically caused by slow storage reads, overloaded shards, or inefficient trace queries.

When pending compaction tasks accumulate faster than they can complete, reads must scan excessive SSTables, driving p99 latency upward and eventually causing timeout exceptions. This silent storage debt compounds over time.

When heap usage climbs above 80-90% without GC recovery, or when GC pause times exceed 500ms, Cassandra cannot process requests during stop-the-world pauses. This manifests as WriteTimeoutException at the application layer despite healthy disk and network.

Rising cassandra_storage_count_hints while all nodes appear up in nodetool status indicates silent network partitions or zombie nodes that accept gossip but fail write requests. This causes eventual consistency drift between replicas.

Cassandra Compaction Backlog Hidden Latencywarning

Rising pending compaction tasks silently degrade read performance by forcing queries to scan more SSTables. This storage debt accumulates invisibly until read latency spikes occur.

JVM Heap Exhaustion Pre-Failure Signalcritical

Heap usage climbing to 80-90% without GC relief indicates impending OutOfMemoryError. Cassandra cannot process requests during heap thrashing, causing cascading application timeouts.

When pending compaction tasks accumulate, read latency degrades because queries must scan many SSTables. This manifests as rising P99 read latency while write latency remains stable, often accompanied by increasing SSTables-per-read counts.

Growing cassandra_storage_count_hints indicates a replica node is unreachable or unable to accept writes, forcing coordinators to buffer writes temporarily. Persistent growth during apparent cluster health suggests network partitions or zombie nodes.

Frequent or long garbage collection pauses appear as query timeouts and high cassandra_read_timeouts / cassandra_write_timeouts, but root cause is JVM memory pressure, not database overload. GC pauses over 500ms translate to client-side timeouts.

Cassandra's SEDA architecture relies on threadpools for async processing. When cassandra_thread_pool_count_blocked_tasks or cassandra_thread_pool_currently_blocked_tasks accumulate, requests queue waiting for threads, causing latency spikes that don't correlate with CPU/disk saturation.

JMX Metrics Collection Gap Blinding Cassandra-Backed Grafana Monitoringcritical

Grafana instances using Cassandra as a backend can experience silent performance degradation when JMX metrics (compaction tasks, heap usage, GC pauses) are not collected. Without JMX visibility, operators miss early warnings of heap exhaustion, compaction backlog, or GC thrashing that manifest as sudden dashboard failures.

JVM Heap Exhaustion Before Cassandra Node Failurecritical

Cassandra nodes running on the JVM can experience heap exhaustion where heap usage climbs to 80-90% and stays elevated without dropping after GC, leading to OutOfMemoryError or node instability. This manifests as timeout errors at the application layer before the node crashes.

When Cassandra's pending compaction tasks (cassandra.compaction.tasks.pending) continually climb without dropping, the cluster is writing faster than it can compact SSTables. This causes read amplification as more SSTables must be scanned per query, directly degrading read latency despite healthy-looking CPU and memory.

Rising cassandra.storage.total_hints.count while all nodes appear healthy indicates writes are being buffered for unreachable replicas. This suggests a network partition, misconfiguration, or 'zombie' node that appears up but isn't accepting writes, causing the coordinator to queue hints indefinitely.

Client-measured latency exceeds service-side cassandra_client_request_read_time/write_time by large margins, indicating network overhead is the bottleneck rather than database processing. This misdiagnosis leads to incorrect remediation efforts.

Frequent deletes create tombstones that must be scanned during reads. When tombstone counts exceed 1000 per read, query latency degrades dramatically despite healthy system resources, often masking as a generic slow query problem.

Old snapshots from backups or schema changes accumulate on disk, silently consuming storage until reaching 90%+ capacity. This blocks compaction and eventually causes node failures with disk-full errors.

When cassandra_key_cache_hit_rate drops significantly below historical baseline, reads bypass cache and hit disk directly, multiplying I/O load and latency. This often correlates with workload changes or cache size misconfiguration.

Excessive cassandra_memtable_switch and cassandra_table_pending_flushes indicate memtables are flushing too frequently, consuming I/O bandwidth and creating write latency spikes. This often results from undersized memtables or write bursts.

Using STCS (Size-Tiered) creates unbounded SSTable growth and prevents fast streaming, while LCS (Leveled) causes excessive write amplification under heavy load. Both reduce effective node density and operational efficiency compared to UCS (Unified) in Cassandra 5.0+.

When Cassandra's internal thread pools (read, mutation, request, compaction) become saturated with blocked tasks, new requests queue up or fail despite available system CPU. This creates latency spikes before visible service degradation.

Low cassandra_key_cache_hit_rate forces Cassandra to read row locations from disk on every query instead of memory, multiplying disk IO. Degraded cache performance appears as read latency increases without corresponding write load changes.

cassandra_dropped_messages represents Cassandra's load-shedding mechanism when requests can't be processed within timeout. This is a leading indicator of cluster overload — messages drop before clients see timeouts, revealing capacity problems earlier than error rates.