High S3 Read Latency Degrading Query Performance
warningIn ClickHouse Cloud or tiered storage deployments, high S3 read latency or error rates cause queries to slow down or fail, especially when filesystem cache hit rate is low.
Monitor S3 read request rate, S3 read wait time, S3 read errors per second, and filesystem cache hit rate. Correlate low cache hit rates with increased query latency. Track DiskS3ReadMicroseconds and S3 error metrics from system.asynchronous_metrics and system.events.
Increase filesystem cache size to improve hit rate. Investigate S3 throttling or region/endpoint issues. Optimize queries to reduce data scanned (better filtering, primary key usage). Consider object storage tier with better latency SLA if consistently slow. Monitor page cache hit rate alongside filesystem cache.