Couchbase insights

Default Linux swappiness settings (typically 60) cause the OS to swap Couchbase memory pages to disk even when RAM is available, creating performance degradation as database operations hit swap instead of memory.

When view map/reduce functions have errors (like incompatible data types with _stats reduce), indexing silently fails but queries with stale=ok or stale=update_after return empty results without error messages. Errors only surface with stale=false queries.

When Transparent Huge Pages (THP) is enabled on Linux, Couchbase can experience severe page allocation delays during memory operations, causing unpredictable latency spikes and performance degradation.

When secondary index updates fall behind (couchbase_index_count_docs_pending grows large), queries with stale=false time out waiting for index updates to complete. This commonly occurs during bulk operations or when index memory is exhausted.

When disk write queue depth (couchbase_by_bucket_vb_*_queue_size) grows large and queue age increases, operations time out as persistence layer falls behind. This often manifests during heavy write workloads or when disk I/O is slow.

When memory pressure forces aggressive ejection (couchbase_by_bucket_evictions, couchbase_by_bucket_vb_*_eject), the cache hit ratio drops dramatically, forcing operations to fetch from disk. This creates a performance spiral as disk fetches are orders of magnitude slower than memory access.

When cross-datacenter replication (XDCR) falls behind, the backlog can consume memory and CPU on the source cluster, impacting primary workload performance. Large item counts in DCP queues indicate replication lag.

When Query service memory (couchbase_query_memory) is exhausted, queries fail or timeout. The Query service has a separate memory quota from data service and can become a bottleneck during complex analytical queries.

When Sync Gateway's shared bucket import falls behind (couchbase_sync_gateway_shared_bucket_import_import_cancel_cas increases), mobile device synchronization is delayed. This creates user-visible lag in collaborative applications.

During cache warm-up, parallel sub-document fetch operations can overwhelm a single Couchbase node, causing cascading timeouts and connection failures. This creates a "snowball effect" where newly spawned instances generate far more requests than the cluster can process.