Artifact Upload Timeout Cascade Failures

warning

latencyUpdated Sep 3, 2025

Large model artifacts (>2GB) consistently timeout during upload to S3 with default timeout settings, causing training pipelines to fail at the final step after hours of computation.

Sources

# MLflow Production Troubleshooting Guide - Fix the Shit That Always Breakstoolstac.com

Technologies:

MLflowSymptoms of this issue are visible in MLflow metrics and logs

mlflow.artifacts.upload_duration_seconds

mlflow.artifacts.upload_timeouts_total

mlflow.artifacts.size_bytes

How to detect:

Monitor artifact upload durations, timeout error rates, artifact sizes, and network throughput. Watch for 'requests.exceptions.ConnectTimeout: HTTPSConnectionPool' errors and failed uploads of large files.

Recommended action:

Increase artifact upload timeout: 'export MLFLOW_ARTIFACT_UPLOAD_TIMEOUT=300'. For very large models, bypass MLflow artifact logging and upload directly to S3, then log only the URI: 'mlflow.log_param("model_uri", "s3://bucket/path")'. Consider multipart uploads for files >100MB.