Token Usage Forecast Drift from Model Changes
warningLangSmith's token usage forecasting assumes stable model behavior. Untagged model version changes (e.g., GPT-4 → GPT-4-turbo, Claude updates) can shift token distributions, invalidating forecasts and triggering false cost alerts.
Monitor langchain_tokens (total), langchain_tokens_prompt, and langchain_tokens_completion for sudden distribution shifts. Cross-reference with metadata grouping (model version tags) to detect untagged model rollouts. Watch for forecast divergence alerts when actual usage deviates >20% from prediction.
Tag all LLM calls with model version in metadata (e.g., {'model_version': 'gpt-4-0613'}). Use LangSmith's grouped monitoring to track per-version token trends. Retrain or reset forecast baselines after model changes. Alert on untagged or newly appearing model versions.