HashiCorp Vault insights

When audit devices fail to write (disk full, permission errors, network unavailable), Vault blocks all requests to prevent audit gaps. This creates complete service outage despite Vault being otherwise healthy.

When vault_core_unsealed drops to 0 or the sum across nodes falls below expected cluster size, Vault has sealed and cannot serve requests. This is a critical failure requiring immediate operator intervention to unseal.

When vault_core_active shows more than one active node or no active nodes, the cluster is experiencing split-brain or leader election failure. This prevents request processing and indicates network partitions or Raft consensus issues.

Unbounded growth in vault_token_count and vault_expire_num_leases indicates applications not revoking tokens or letting leases expire naturally, leading to memory pressure and slower revocation operations. This is a common misconfiguration pattern.

High P99 latency (histogram_quantile(0.99, vault_core_handle_request_duration_seconds)) indicates a subset of requests experiencing severe delays, often from storage backend issues, token/policy evaluation overhead, or resource exhaustion.

Sustained growth in vault_go_memstats_heap_allocated_size and vault_go_memstats_heap_objects without corresponding decreases indicates memory leak or excessive in-memory state (tokens, leases, cache). Can lead to OOM and seal events.

High frequency or slow completion of rollback operations (vault_route_rollback_*, vault_rollback_attempt_*) indicates failed partial writes accumulating faster than cleanup, potentially from application errors or storage instability. Can cause performance degradation.

Increasing vault_wal_persistwals and vault_wal_loadwal metrics indicate write-ahead log operations are slow or backing up, suggesting disk I/O saturation or fsync latency issues. This directly impacts write performance in Raft-backed clusters.

Growing vault_expire_job_manager_queue_length_datadog indicates lease expiration and revocation operations are backing up, typically from high lease creation rate, slow storage backend, or insufficient processing capacity. Can lead to delayed revocations and security exposure.