MongoDB insights

Operations waiting for database-level locks indicate contention, often from administrative operations (dropDatabase, reIndex), or in older MongoDB versions, from database-level write concerns.

Chunks exceeding the configured chunk size (default 64MB) become jumbo chunks that cannot be moved by the balancer, preventing even distribution of data across shards and creating persistent hot spots.

Foreground index builds acquire exclusive locks blocking all operations on the database/collection. Background index builds (deprecated in 4.2+) still impact performance. Large index builds can cause significant operational disruption.

Operations spending significant time waiting to acquire locks indicates lock contention, often from long-running operations, bulk writes, or index builds blocking concurrent operations.

When working set exceeds available RAM, frequent page evictions occur as WiredTiger evicts pages from cache, causing disk I/O spikes and query latency increases. Cache thrashing severely degrades performance.

WiredTiger storage engine uses a ticket system to limit concurrent read/write operations. Ticket exhaustion causes operations to queue, increasing latency even when CPU and I/O are available.

Failed or interrupted chunk migrations can leave orphaned documents on the source shard. These documents consume disk space, aren't served by queries, and indicate migration reliability issues.

Excessive concurrent sessions, particularly from multi-document transactions, can exhaust session limits causing new operations to fail. Long-running transactions hold sessions open, preventing cleanup.

Secondaries spending excessive time reading from oplog or experiencing slow oplog queries indicates network issues, oplog access contention, or secondary resource constraints affecting replication throughput.

Queries with sort operations that cannot use indexes require in-memory sorting, consuming memory and CPU. When sorted data exceeds 32MB memory limit, queries fail or spill to disk (MongoDB 4.4+), severely degrading performance.

Frequent cursor timeouts indicate queries returning large result sets that aren't fully iterated, long-running queries exceeding cursor timeout (default 10 minutes), or client applications failing to close cursors properly.

MongoDB process memory (resident set size) approaching system memory limits causes OS-level swapping, page faults, and severe performance degradation as the system struggles to manage memory pressure.

Reaching MongoDB's concurrent connection limits causes 'connection refused because too many open connections' errors, freezing application operations and causing timeouts.

Secondary nodes fall behind primary during high write volume, causing stale reads and potential failover delays. Often indicates under-provisioned secondaries or inefficient write operations.

Multiple concurrent updates to the same document (hot document pattern) or large embedded arrays cause WiredTiger write conflicts, serializing operations and creating waiting queues even with adequate hardware.

Using monotonically increasing values (timestamps, ObjectIDs) as shard keys concentrates all new writes on a single shard, leaving other shards idle and creating a write bottleneck.

Frequent leader elections without actual server failures indicate resource exhaustion or network issues preventing timely heartbeats, causing application freezes and timeout errors.

Poorly ordered aggregation pipeline stages, especially sorting before filtering or lookup operations without indexes, cause excessive in-memory processing and slow query performance.

Queries examining many more documents than returned indicate inefficient access patterns, missing indexes, or overly broad filters causing unnecessary processing work.

Queries performing COLLSCAN (full collection scans) instead of IXSCAN (index scans) cause excessive document examination, consuming IOPS and blocking WiredTiger cache, especially critical in large collections.

Insufficient oplog size causes replica set members to fall too far behind, requiring full resynchronization when they cannot catch up within the oplog window timeframe.