Metrics Server Failure in Meshed Namespaces

warning

reliabilityUpdated Feb 23, 2023

Kubernetes metrics-server stops collecting pod metrics after Linkerd injection due to cAdvisor bug with CRI runtime. Terminated linkerd-init containers continue producing metrics, causing metrics-server to ignore entire pods.

Sources

Kubernetes metrics-server not working with Linkerdstackoverflow.com

Technologies:

LinkerdThe root cause of this issue originates in Linkerd

KubernetesSymptoms of this issue are visible in Kubernetes metrics and logs

How to detect:

Monitor for 'Metrics not available for pod' errors when running 'kubectl top pod' in meshed namespaces, while metrics-server functions normally in non-meshed namespaces. No errors appear in metrics-server logs.

Recommended action:

Use PodAndContainerStatsFromCRI feature flag to switch from cAdvisor to internal CRI stats provider, or change container runtime. This is a known cAdvisor bug where init containers incorrectly report metrics post-termination.