Google GKE insights

When the GKE control plane cannot communicate with nodes (e.g., due to SSH tunnel issues), metric and log collection fails, causing complete observability blackout for affected clusters.

Alert Incidents Not Matching GKE Resources Due to Missing Hierarchy Labels

When alerting policies aggregate metrics across multiple clusters without including namespace, cluster, and location in the Group By field, incidents cannot be uniquely associated with specific services or environments in the GKE dashboard.

The gke-metrics-agent DaemonSet can crash with OOMKilled when default memory allocation is insufficient for high-cardinality metric environments, causing gaps in system and workload metrics.

Missing Metrics Due to IAM Permission Gaps

When node service accounts lack roles/container.defaultNodeServiceAccount, the gke-metrics-agent experiences 401 errors and fails to write metrics to Cloud Monitoring, resulting in blind spots in observability.

When node log volume exceeds ~100 KiB/s, the Fluent Bit logging agent becomes CPU-saturated and starts dropping logs, leading to incomplete observability and missed incidents.