Ceph insights

When OSDs report slow operations (requests taking >30s by default), it indicates disk I/O bottlenecks, journal issues, or network problems affecting cluster performance. This is one of the most common Ceph performance issues.

When OSDs reach nearfull (default 85%) or full (default 95%) thresholds, Ceph begins throttling operations and can trigger rebalancing, severely degrading performance. Full OSDs prevent all writes and can cause cluster-wide unavailability.

OSDs repeatedly marking themselves up/down (flapping) due to failed heartbeats indicate underlying network problems, overloaded OSDs, or failing hardware. This causes continuous recovery operations and severely impacts performance.

PGs not in 'active+clean' state (stale, degraded, inconsistent, down) indicate data availability or consistency issues. These states directly impact performance and data safety, with different states requiring different remediation approaches.

When recovery operations (rebalancing, backfill) occur due to OSD failures or reweighting, they compete with client I/O for disk and network resources. High recovery rates indicate ongoing data movement that degrades client-facing performance.

When data distribution across OSDs is significantly uneven (high variance in ceph_osd_pct_used), some OSDs become hotspots handling disproportionate load while others remain underutilized. This reduces overall cluster performance and accelerates filling of busy OSDs.

Monitor clock drift exceeding 50ms (default mon_clock_drift_allowed) causes monitors to mark each other as down, breaking quorum and preventing cluster operations. This manifests as monitors being out of quorum despite being healthy.

When the monitor store (RocksDB database) grows excessively large (default warning at 15GB), monitor queries slow down, potentially causing client timeouts and leader election delays. If /var partition fills completely, monitors terminate.