Envoy Proxy insights

Service Mesh Request Error Rate Spike

Rising 5xx error rates in Envoy upstream metrics indicate service-side failures before health checks mark services as down. This provides early warning of application degradation.

Envoy Connection Failure Pattern

Rising envoy_cluster_upstream_cx_connect_fail indicates network path issues or upstream service unavailability before requests even attempt, revealing infrastructure-level problems.

When Envoy cannot connect to upstream services, it returns 503 errors with response flags like UC (upstream connection failure) or UH (no healthy upstream). This often indicates backend health issues or network problems that can cascade across the mesh.

Each HTTP filter in Envoy's processing chain adds latency. Long filter chains with unnecessary or poorly configured filters compound per-request overhead, especially impacting p95/p99 latency in high-throughput systems.

When Envoy's active health checks target non-existent endpoints or use incorrect protocols, all upstream hosts are marked unhealthy (failed_active_hc flag), causing complete service outages despite healthy backends.

Multiple listeners attempting to bind the same port without proper filter chain matching cause Envoy startup failures or runtime errors. This prevents traffic from being served on affected ports.

Envoy's response flags (UH, UF, UT, NR, DC, LH, RL, etc.) in access logs provide precise diagnostic information for request failures. Each flag points to specific failure modes that require different remediation strategies.

Envoy uses short-lived certificates (2-3 days default) for service mesh mTLS. When certificates expire before rotation, TLS handshake failures occur, breaking service-to-service communication with SSLV3_ALERT_CERTIFICATE_EXPIRED errors.

504 errors in Nginx access logs indicate backend API services are timing out. Extracting and counting affected URLs reveals which Chef Infra Server services (erchef, bifrost) are overloaded or failing.

The ext_authz filter's failure_mode_allow setting determines whether requests proceed (fail-open) or are blocked (fail-closed) when the authorization service is unavailable. Misconfiguration creates either security gaps or availability failures depending on business requirements.

When Envoy receives stale IP addresses from EDS (Endpoint Discovery Service), it attempts connections to non-existent hosts, resulting in 113 (EHOSTUNREACH) errors and failed requests until configuration updates.