Drive Performance Outlier Detection

critical

Resource ContentionUpdated May 2, 2025

In distributed MinIO deployments, a single slow or failing drive can bottleneck all write operations due to erasure coding requirements, but standard monitoring may not surface per-drive latency.

Sources

Metrics and Loggingmin.io

Low Level Performance Testing for Object Storage - MinIO Blogblog.min.io

MinIO Cluster Benchmark Methods and Toolswww.intel.com

Technologies:

MinIOSymptoms of this issue are visible in MinIO metrics and logs

minio.drive.read_throughput_bytes

minio.drive.write_throughput_bytes

minio.drive.read_latency_seconds.p99

minio.drive.errors.total

How to detect:

Use dperf or fio to baseline individual drive throughput. Alert when any single drive shows >30% lower throughput than peers, or when drive latency (p99) exceeds 2x cluster median. Monitor for drives with increasing error counts or timeouts in MinIO logs.

Recommended action:

Run 'mc admin speedtest drive' or use dperf to identify slow drives. Check drive health with smartctl. Verify drive is properly connected (check NUMA/PCIe topology with hwloc-ls). Replace failing drives immediately. Use 'mc support diag' to collect detailed drive diagnostics for MinIO support analysis.