HashiCorp Consul insights

Consul Raft commit times above 200ms indicate network or disk performance problems that affect cluster write operations. This directly impacts service registration updates and configuration changes.

Service Mesh Request Error Rate Spike

Rising 5xx error rates in Envoy upstream metrics indicate service-side failures before health checks mark services as down. This provides early warning of application degradation.

When consul_raft_leader_lastcontact exceeds 200ms, the Raft leader is taking too long to contact followers, indicating network issues or server overload that threatens cluster stability.

Memberlist health score elevation or degraded probe counts indicate gossip protocol issues that can lead to incorrect failure detection and cluster instability.

consul_health_service_checks_critical directly maps services that Consul will stop routing traffic to. This is the most actionable health metric as it indicates services currently unavailable for discovery.

Spikes in consul_rpc_request_error precede broader cluster issues as RPC is how Consul agents communicate. This provides early warning before user-visible problems occur.

Envoy Connection Failure Pattern

Rising envoy_cluster_upstream_cx_connect_fail indicates network path issues or upstream service unavailability before requests even attempt, revealing infrastructure-level problems.

Frequent leadership changes (rate of consul_raft_state_leader or consul_raft_state_candidate changes) indicate network instability between servers that threatens cluster write availability.

High consul_dns_stale_queries rate means Consul clients can't reach servers for fresh data, serving potentially outdated service discovery information that can route traffic to wrong endpoints.

Elevated consul_runtime_gc_pause_ns metrics indicate memory pressure causing GC pauses that delay Consul operations, potentially triggering false failure detection in Raft or gossip protocols.