LangSmith insights

Missing or incorrectly routed traces often result from cached environment variables in development environments, particularly Jupyter notebooks. This prevents proper trace collection and project routing.

Unexpected increases in token usage and costs indicate inefficient prompts, excessive context, or unoptimized model calls. Cost control is critical for LLM application economics.

When testing different model providers, prompts, or configurations, monitoring enables data-driven decisions by comparing latency, cost, and quality metrics across variants.

Slow or failing tool calls (run type 'tool') indicate issues with external dependencies like APIs, databases, or search services. Tool latency directly impacts end-to-end agent performance.

Token consumption and API costs escalate without visibility or budget controls, leading to unexpected billing and inefficient resource utilization.

Model Performance Variance Across Versions

Different model providers or configurations exhibit significantly different latency and quality characteristics, but lack of grouped monitoring prevents data-driven model selection.

High error rates in LLM applications indicate failures from model provider rate limits, API issues, or application logic errors. Proactive alerting prevents user-facing failures and enables rapid response.

Average run latency increases indicate performance bottlenecks from slow model providers, inefficient tool calls, or infrastructure issues. Latency spikes directly impact user experience.

Declining average feedback scores signal quality degradation from prompt changes, model updates, or data drift. User satisfaction metrics are critical for LLM application reliability.

Model Provider Rate Limit Detection

Errors from model provider rate limits (e.g., RateLimitExceeded) cause application failures and poor user experience. Different providers have varying latency and throughput characteristics.

LLM applications exhibit elevated error rates (API failures, rate limits, tool call failures) without proactive notification, leading to degraded user experience before detection.

Application response times increase significantly (P95/P99 spikes) without triggering alerts, causing poor user experience and potential SLA violations.

LLM output quality degrades (hallucinations, incorrect answers, poor user satisfaction) but goes undetected due to lack of continuous feedback monitoring.

Time-to-first-token (TTFT) affects perceived responsiveness of streaming LLM applications. High TTFT impacts user experience even if total latency is acceptable.

Applications using streaming LLM responses experience poor time-to-first-token performance, impacting perceived responsiveness, but lack specific monitoring for streaming metrics.