Network Bandwidth Saturation

warning

Resource ContentionUpdated Dec 30, 2024

Network send/receive rates approaching hardware limits cause query slowdowns, especially for distributed queries or replication in multi-node clusters.

Sources

Common issues you can solve using advanced monitoring ...clickhouse.com

Technologies:

ClickHouseSymptoms of this issue are visible in ClickHouse metrics and logs

clickhouse.network.receive_bytes_per_sec

clickhouse.network.send_bytes_per_sec

clickhouse.metric.ConcurrentNetworkConnections

How to detect:

Monitor NetworkReceiveBytesPerSecond and NetworkSendBytesPerSecond from system.asynchronous_metrics. Alert when sustained traffic exceeds 80% of network interface capacity.

Recommended action:

Upgrade network interfaces or switch fabric, implement query result compression, optimize distributed queries to reduce data shuffling, or colocate related data on same nodes to minimize network transfers.