Helm deployment regenerates basic auth secret causing 401 cluster failures

critical

configurationUpdated Oct 8, 2024

Sources

Node Desync and Cluster Inconsistencies After OOM on Weaviate-0 - Support - Weaviate Community Forumforum.weaviate.io

Technologies:

Weaviatesubject

ArgoCDThe root cause of this issue originates in ArgoCD

How to detect:

When using Helm (especially with ArgoCD), the weaviate-cluster-api-basic-auth secret is regenerated on each deployment because ArgoCD doesn't support the helm lookup function. This causes inter-node authentication to fail with HTTP 401 errors and breaks cluster communication.

Recommended action:

Configure ArgoCD to ignore differences in the weaviate-cluster-api-basic-auth secret by adding ignoreDifferences with jsonPointers for /data/username and /data/password. Alternatively, manually manage the secret outside of Helm or use a Helm deployment method that supports the lookup function.