MLflow

MLflow insights

Open SourceVersions: [current]21 metrics
MLflow
Experiment Search Performance Degradation with Complex Filterswarning

MLflow experiment search API becomes extremely slow (>30 seconds) when using complex filter expressions with multiple tag/param/metric conditions, especially on large experiment sets. This is caused by inefficient query generation that performs multiple sequential scans instead of leveraging indexes and set operations.

28d ago▸
MLflow
Artifact Download Performance Bottleneck for Model Servingwarning

Model serving systems experience high startup latency and deployment failures when downloading large model artifacts (>1GB) from MLflow artifact storage, particularly when models are stored in S3 with default download timeouts or when artifact storage is in a different region from serving infrastructure.

28d ago▸
MLflow
Model Registry Metadata Bloat from Excessive Versioninginfo

Teams registering hundreds or thousands of model versions without cleanup create metadata bloat in the model registry, degrading UI performance and making model discovery difficult. Each model version stores full metadata, tags, and lineage information, leading to large database tables and slow queries when listing or searching models.

28d ago▸
MLflow
Artifact Storage Cost Explosioncritical

S3/cloud storage costs explode due to MLflow storing all artifacts forever by default, including large training datasets and intermediate model checkpoints logged with every experiment.

toolstac.com
6mo ago▸
MLflow
Memory Leak in MLflow Server Processcritical

MLflow tracking server exhibits unbounded memory growth over time, reaching 8-20GB RAM consumption even with moderate experiment counts (10,000+). Memory growth correlates with cached experiment metadata, model registry data, and internal state that is never evicted, eventually leading to OOM kills and service disruption.

toolstac.com
6mo ago▸
MLflow
Artifact Upload Timeout Cascade Failureswarning

Large model artifacts (>2GB) consistently timeout during upload to S3 with default timeout settings, causing training pipelines to fail at the final step after hours of computation.

toolstac.com
6mo ago▸
MLflow
Async Trace Logging Queue Overflow in High-Traffic Productionwarning

When production GenAI traffic exceeds async logging queue capacity (default 1000 traces), new traces are silently discarded, creating observability blind spots during peak load or incidents.

mlflow.orgmlflow.org
1y ago▸
MLflow
LLM Judge Evaluation Cost Spiralwarning

Automatic quality evaluation using LLM judges on 100% of production traces creates unsustainable API costs, especially for high-volume GenAI applications with thousands of daily requests.

mlflow.orgmlflow.org
1y ago▸