Network Asymmetry Between Nodes
warninglatencyUpdated Jul 11, 2023
Uneven network performance between MinIO nodes (e.g., one node on 10Gbps while others on 25Gbps) creates replication and erasure-coding bottlenecks that appear as slow writes but are actually network-limited.
Sources
Technologies:
How to detect:
Measure inter-node bandwidth with hperf (expect 50-75% of nominal NIC speed). Alert when any node-to-node link shows <50% of expected throughput. Cross-reference with minio.api.request_duration_seconds.p99 spikes during high write load.
Recommended action:
Run hperf between all node pairs to identify asymmetric links. Check NIC configuration with ethtool (expect 25/100Gbps). Verify switch port configuration. Use separate networks for client traffic and inter-node traffic if possible. Monitor network saturation during peak write workloads.