API Error Spike Without Root Cause Context

warning

reliabilityUpdated Jul 24, 2024

Increases in S3 API error rates (4xx/5xx) indicate client misconfigurations or service degradation, but without correlated tracing or audit logs, operators cannot distinguish between application bugs, policy issues, or infrastructure failures.

Sources

A Closer Look: MinIO Observabilityblog.min.io

Powerful Perspective: Introducing MinIO Observabilityblog.min.io

Observability | MinIO AIStorwww.min.io

Technologies:

MinIOSymptoms of this issue are visible in MinIO metrics and logs

minio.api.errors.4xx.total

minio.api.errors.5xx.total

minio.api.request_duration_seconds.p99

minio.audit.access_denied.total

How to detect:

Alert on minio.api.errors.4xx.total or minio.api.errors.5xx.total rate increase >3x baseline. Correlate with OpenTelemetry trace data (if enabled) to identify error distribution across API operations (GetObject, PutObject, ListBucket). Check audit logs for AccessDenied patterns.

Recommended action:

Enable OpenTelemetry tracing with 'mc admin trace' or configure OTLP export to correlate errors with request paths. Review audit logs for policy evaluation failures. Use 'mc admin trace --call' to capture live API calls during error spikes. Check for client SDK version issues or retry storms.