Arize Phoenix insights

LLM Response Latency and Token Cost Correlation

High token usage in LLM spans directly correlates with increased latency and cost, particularly in multi-step agent workflows. Spans with >5,000 prompt tokens can significantly impact overall trace duration and monthly costs.

Runtime Exception and Rate Limiting Events

LLM applications may encounter runtime exceptions or rate-limiting from API providers, causing cascading failures and degraded user experience. These exceptions are critical to track as they directly impact service availability.

Tool Call Accuracy and Usage Patterns

AI agents may misuse tools by calling incorrect functions, passing invalid parameters, or executing tools in inefficient sequences. This manifests as increased latency, unnecessary API calls, and poor task completion rates.

As trace volume grows in Phoenix, project page load times increase significantly. Performance degrades with tens of thousands of traces, especially when traces contain large numbers of spans (>100) with significant token counts.

Changes to prompt templates or variables may inadvertently degrade model performance. Without tracking, teams cannot correlate performance changes with specific prompt iterations, making it difficult to identify and roll back problematic changes.