Ray Telemetry High-Cardinality Cost Explosion
warningAttaching Kubernetes pod-level attributes (k8s.pod.id, k8s.node.ip) to Ray metrics dramatically increases cardinality and observability costs, especially in autoscaling environments where pods are ephemeral.
Monitor observability backend costs correlated with Ray cluster autoscaling events (ray_cluster_active_nodes changes). Track unique time series count increasing when k8s.pod.id is attached to high-frequency metrics like ray_serve_count_http_requested or ray_scheduler_tasks. Detect when pod IDs in telemetry data reference terminated pods (cross-reference with k8s pod lifecycle).
Use k8sattributes processor selectively: attach pod-level labels (k8s.pod.id, k8s.pod.ip) only to traces and logs, not metrics. For metrics, limit to low-cardinality attributes like k8s.namespace.name, k8s.deployment.name, k8s.node.name. Configure extract.metadata in k8sattributes to exclude high-cardinality fields from metrics pipeline. Implement separate processor configs for metrics vs traces/logs pipelines. Monitor observability cost trends when autoscaling triggers and adjust attribute strategy accordingly.