LlamaIndex Embedding Token Inefficiency
warningLlamaIndex embedding operations consume excessive tokens due to redundant document processing, lack of caching, or inefficient chunking strategies, increasing costs and latency.
Monitor llama_index.embedding.tokens for abnormally high values relative to llama_index.embedding.requests. Alert when average tokens per embedding request (embedding.tokens / embedding.requests) exceeds 2x baseline or when embedding.tokens.total shows unexplained growth. Track llama_index.embedding.duration to identify processing bottlenecks.
1. Investigate: Calculate tokens per request ratio. Identify document types or sources with highest token consumption. Check if embeddings are being regenerated unnecessarily. 2. Diagnose: Review chunking strategy (chunk size, overlap). Verify embedding cache is enabled and hit rate is acceptable. Check for duplicate document processing. Analyze token distribution across embedding.tokens.prompt vs completion. 3. Remediate: Optimize chunk size (512-1024 tokens is typical for semantic search). Implement embedding caching with document content hash as key. Enable incremental indexing to avoid re-embedding unchanged documents. Use batch embedding APIs to reduce overhead. 4. Prevent: Set budget alerts on embedding.tokens with daily thresholds. Dashboard tokens per document type. Monitor cache hit rate (custom metric). Implement embedding cost estimation in ingestion pipeline to preview costs before bulk operations.