Embedding Provider Rate Limiting and Quota Exhaustion
warningChroma relies on embedding functions (OpenAI, Cohere, HuggingFace, custom) to generate vectors. Rate limits, quota exhaustion, or API failures from embedding providers cause upsert/query operations to fail or experience severe latency. This is especially problematic during bulk ingestion or high query concurrency.
Upsert or query operations fail with embedding provider errors (rate limit, quota exceeded, timeout). Error rate spikes correlate with high embedding request volume. Latency increases due to provider throttling or retries. Operations succeed after backoff/retry or quota reset.
1. Monitor provider status: Check embedding provider status pages and dashboards. Review quota usage and rate limit thresholds. 2. Implement retry with backoff: Add exponential backoff for rate limit errors (respect Retry-After headers). Queue failed operations for retry. 3. Rate limit client-side: Throttle embedding requests to stay below provider limits. Implement token bucket or leaky bucket rate limiting. 4. Batch efficiently: Use provider batch APIs where available to maximize throughput within rate limits. 5. Cache embeddings: Cache embedding results for repeated text to reduce API calls. Use content-based hashing for cache keys. 6. Provision higher tier: Upgrade embedding provider plan for higher rate limits and quotas if needed. 7. Fallback provider: Implement fallback to alternative embedding provider if primary fails. Note: requires consistent embedding space. 8. Monitor: Track embedding API latency, error rates, quota usage. Alert on rate limit errors or quota approaching limit.