ClickHouse Disk Space Exhaustion in Self-Hosted Observability

critical

Resource ContentionUpdated Feb 23, 2026

Self-hosted LangSmith observability stacks crash when ClickHouse runs out of disk space during async trace insertion, manifesting as NOT_ENOUGH_SPACE errors that prevent trace ingestion.

Sources

Troubleshooting - Docs by LangChaindocs.langchain.com

Technologies:

DockerSymptoms of this issue are visible in Docker metrics and logs

ClickHouseThe root cause of this issue originates in ClickHouse

LangChainLangChain metrics correlate with this issue and help confirm diagnosis

How to detect:

Monitor for DB::Exception 'NOT_ENOUGH_SPACE' errors during WaitForAsyncInsert operations. Watch ClickHouse volume usage approaching capacity. Track trace ingestion rate spikes that correlate with storage growth.

Recommended action:

Increase ClickHouse PVC/volume size before reaching capacity. For Kubernetes: verify storage class allows expansion, patch PVC size, delete statefulset with --cascade=orphan, reapply helm chart. For Docker: allocate more space to volume/mount point. Implement TTL-based data retention policies to prevent unbounded growth.