API Server Latency from Admission Webhooks

warningProactive Health

Identify admission webhooks causing high API server request latency and cluster-wide slowdowns.

Prompt: “Our kubectl commands and pod deployments are really slow, and I suspect admission webhooks are adding latency — how do I identify which webhooks are the bottleneck and whether they're timing out?”

Agent Playbook

When an agent encounters this scenario, Schema provides these diagnostic steps automatically.

When diagnosing slow kubectl commands and pod deployments caused by admission webhooks, start by confirming elevated API server request latency, then identify which specific webhooks are slow using per-webhook duration metrics. Check for webhook timeouts and failures before investigating deeper infrastructure issues like webhook endpoint health, etcd performance, or API server overload.

1Verify API server request latency is actually elevated

Before blaming webhooks, confirm the symptom by checking `apiserver_request_duration_seconds` at p95 and p99. Normal p99 latency should be under 1 second; if you're seeing sustained p99 >2-3s, you have a real problem. This metric is tracked in the `high-latency-from-slow-api-server-or-scheduler` insight and gives you a baseline before diving into webhook-specific diagnostics.

High Latency from Slow API Server or Scheduler

2Identify which admission webhooks are adding latency

Use `apiserver_admission_webhook_admission_duration_seconds` broken down by webhook name to see per-webhook latency. Look for webhooks with p95 >500ms or p99 >1s—these are your bottlenecks. Common culprits include OPA/Gatekeeper policy evaluation, Istio sidecar injection, and custom validation webhooks. Focus on the webhooks called most frequently on mutating vs. validating paths.

Kubernetes Admission Controller Timeout

3Check for admission webhook timeouts

Look at `apiserver_admission_webhook_rejection_count` filtered by reason=timeout to see if webhooks are exceeding their timeout threshold (typically 10-30s depending on configuration). Timeouts are critical because they block admission requests completely. If you see timeouts on OPA Gatekeeper specifically, the policy evaluation is likely too complex—see the `kubernetes-admission-controller-timeout` insight for optimization guidance on using linear-time Rego and indexed statements.

Kubernetes Admission Controller Timeout

4Check webhook endpoint health and network latency

Verify the webhook service itself isn't overloaded by checking `kubernetes_cpu_usage` and `kubernetes_memory_usage` for webhook pods—if CPU is pegged at limit or memory is high, the webhook can't respond quickly. Also check `kubernetes_network_errors` between API server nodes and webhook endpoints; packet loss or network issues add unpredictable latency that manifests as slow admissions.

kubernetes_cpu_usagekubernetes_memory_usagekubernetes_network_errors

5Investigate whether etcd latency is contributing

Slow etcd can cause overall API server latency that looks like webhook delays but isn't. Check etcd-specific `apiserver_request_duration_seconds` metrics and disk latency—if etcd p99 >100ms, that's your real problem. The `api-server-request-latency-impacting-kubernetes-operations` insight covers this: when etcd is slow, all API requests (including webhook callbacks) get delayed, causing kubectl timeouts and scheduler lag.

API Server Request Latency Impacting Kubernetes Operations

6Check for API server overload or rate limiting

If `apiserver_request_total` shows elevated 429 (rate limiting) or 500 errors, the API server itself may be the bottleneck, not the webhooks. Check API server pod `kubernetes_cpu_usage` and `kubernetes_memory_usage`—at high scale (many webhooks, large clusters), you may need to scale API servers horizontally or optimize client request patterns. The `api-server-rate-limiting-active` and `api-server-overload-at-scale` insights cover this scenario.

API Server Rate Limiting Active API server overload at high scale causes HTTP 500 errors kubernetes_cpu_usagekubernetes_memory_usage