vllm:request_time_per_output_token_seconds

Summary

In vLLM v0.23.0, under the PrometheusStatLogger metrics publisher with V1 metrics, vllm:request_time_per_output_token_seconds is registered as a Prometheus histogram. Its registration declares 19 literal histogram bucket boundaries ranging from 0.01 to 80.0. It is registered with the label names model_name and engine. For each finished request it observes a single value: the mean decode interval, that is, the decode time divided by the number of output-token intervals after the first token. In this version it is computed from the first-to-last-token decode time divided by num_generation_tokens minus one, yielding zero when no such interval exists. Interpretively, it is a request-weighted TPOT distribution and therefore does not describe the same population as the token-weighted inter_token_latency_seconds. Use it to gauge the per-request decode experience, comparing distributions under matched output lengths rather than substituting aggregate tokens per second.

Version history

• The only version documented here is v0.23.0, where vllm:request_time_per_output_token_seconds is registered as a Prometheus histogram under the PrometheusStatLogger V1 metrics path with the label names model_name and engine, and with 19 literal bucket boundaries spanning 0.01 to 80.0.
• In that same version it records one observation per finished request, namely the mean decode interval computed as first-to-last-token decode time divided by the output-token intervals after the first token (num_generation_tokens minus one), yielding zero when no such interval exists.
• Still in v0.23.0, the resulting distribution is request-weighted TPOT and therefore is not the same population as the token-weighted inter_token_latency_seconds.
• Accordingly, v0.23.0 guidance is to use it for per-request decode experience and to compare distributions under matched output lengths rather than substituting aggregate tokens per second.

Identity

Prometheus type

histogram

Labels

model_nameengine

Buckets

19 boundaries · 0.01 → 80

Unit

Not recorded — vLLM registration code has no structured unit field, and units are not inferred from name suffixes

Semantic coverage

Metric knowledge (7)

Related Use Cases (1)

• #35048: Aggregate throughput can hide experience regressions

Availability