ClickHouse insights

Self-Hosted Deployment Health Blind Spotscritical

Self-hosted LangSmith instances experience infrastructure issues (disk space, resource constraints, pod failures) that are not detected through application-level monitoring alone.

ClickHouse Disk Space Exhaustion in Self-Hosted Observabilitycritical

Self-hosted LangSmith observability stacks crash when ClickHouse runs out of disk space during async trace insertion, manifesting as NOT_ENOUGH_SPACE errors that prevent trace ingestion.

Supabase and ClickHouse Connection Exhaustioncritical

AgentOps backend relies on Supabase (PostgreSQL) for authentication and ClickHouse for analytics. Misconfigured connection pools or lack of connection reuse can exhaust database connections under load, causing agent traces and metrics to fail silently.

ClickHouse Disk Space Exhaustioncritical

ClickHouse running out of disk space causes NOT_ENOUGH_SPACE errors and prevents trace ingestion. This is a critical infrastructure failure for self-hosted deployments.

High p95/p99 query duration indicates tail latency issues from inefficient queries, missing indexes, or resource contention, degrading user experience.

Memory usage approaching system limits can cause query failures, crashes, or degraded performance, especially during large aggregations or joins.

Growing merge queue depth indicates data ingestion is outpacing merge capacity, leading to increased part counts and eventual performance degradation.

High-frequency small inserts create numerous small parts that degrade query performance and overwhelm the merge system, violating ClickHouse best practices.

Elevated failed query rate indicates systemic issues like resource exhaustion, configuration errors, or application bugs affecting database stability.

High disk I/O wait times slow query execution and merge operations, often caused by undersized storage, sequential I/O patterns, or competing workloads.

ZooKeeper connectivity problems or high request latency disrupt replication coordination, causing replica lag, failed writes, or split-brain scenarios.

Replication Lag Causing Data Inconsistencycritical

In replicated table setups, replication lag increases, causing reads from replicas to return stale data and potentially triggering failover delays or data loss on node failure.

Application-level OpenTelemetry traces are disconnected from ClickHouse server-side query logs, making it difficult to correlate slow application requests with specific ClickHouse query performance. This creates blind spots in distributed tracing.

High-cardinality GROUP BY operations or large JOINs exceed per-query or per-user memory limits, causing query failures with memory limit exceptions.

Queries grouping by high-cardinality columns (millions of unique values like session_id) fail or time out due to excessive memory usage or hash table overhead, especially without external aggregation configured.

Queries exceeding memory limits cause 'Memory limit exceeded' errors, killing queries mid-execution. This often results from high-cardinality GROUP BY operations, large JOINs, or missing LIMIT clauses on exploratory queries.

Queries fail to leverage partition pruning when filters use functions on partition columns, causing full table scans. This dramatically increases bytes read and query latency, especially on time-series data partitioned by date.

Queries filtering on columns not in ORDER BY key prefix perform full granule scans instead of using primary index, reading far more data than necessary. This manifests as high read_bytes relative to result size.

CPU-intensive query operations (complex aggregations, string parsing, decompression) saturate CPU resources, causing query queuing and increased latency. This is identifiable through high UserTimeMicroseconds in query profiles.

Declining compression ratios indicate data characteristics changing or suboptimal codec settings, increasing storage costs and I/O overhead.

High S3 Read Latency Degrading Query Performancewarning

In ClickHouse Cloud or tiered storage deployments, high S3 read latency or error rates cause queries to slow down or fail, especially when filesystem cache hit rate is low.