Schema.ai#35048: Aggregate throughput can hide experience regressions
Equal tokens per second does not imply equal TTFT, queueing, or decode pace
Reported environment
Same model and hardware across an upgrade with similar throughput but worse TTFT and output rate.
The nuance
Token and iteration metrics describe system work; TTFT queue ITL TPOT and E2E describe distinct parts of per-request experience.
Expected behavior
Return both families and require a fixed workload/version-matched comparison before assigning cause.
What Schema establishes
Schema supplies the decomposition; root cause requires controlled A/B or commit bisect.
Runtime evidence still needed
- Fixed matched workload environment and dependencies
- Same-window throughput and per-request latency distributions
Try the questions (2)
How can throughput stay constant while inter-token latency worsensrun →Which metrics separate aggregate efficiency from user experiencerun →Grounding claims
- In vLLM v0.23.0,
vllm:time_to_first_token_secondsis registered as a Prometheus histogram. - In vLLM v0.23.0,
vllm:inter_token_latency_secondsis registered as a Prometheus histogram. - In vLLM v0.23.0,
vllm:request_time_per_output_token_secondsis registered as a Prometheus histogram. - In vLLM v0.23.0,
queued_timeis computed asreq_stats.scheduled_ts-req_stats.queued_ts; the source comment describes the queued interval as from firstQUEUEDevent to firstSCHEDULED. - In vLLM v0.23.0,
vllm:e2e_request_latency_secondsis registered as a Prometheus histogram. - In vLLM v0.23.0,
vllm:prompt_tokensis registered as a Prometheus counter. - In vLLM v0.23.0,
vllm:generation_tokensis registered as a Prometheus counter. - In vLLM v0.23.0,
vllm:iteration_tokens_totalis registered as a Prometheus histogram.