vLLM v0.23.0 · Version scoped

vllm:iteration_tokens_total

Histogram of the number of tokens per engine step.

MetricPrometheusverified Sep 29

Summary

In vLLM v0.23.0, under the PrometheusStatLogger metrics publisher with V1 metrics, vllm:iteration_tokens_total is registered as a Prometheus histogram. Its registration in that same v0.23.0 PrometheusStatLogger/V1 metrics path declares the label names model_name and engine. The v0.23.0 registration also declares 13 literal histogram bucket boundaries, ranging from 1.0 to 16384.0. In v0.23.0 with PrometheusStatLogger and V1 metrics, the metric observes one histogram sample per engine step, whose value is the computed prompt tokens plus generated tokens. These samples are populated from iteration-level work after each IterationStats update in v0.23.0. Despite the word "total" in its declared name, in v0.23.0 the metric describes the distribution of batch/step work rather than acting as a cumulative token counter. Accordingly, in v0.23.0 use its distribution to inspect batching, and do not calculate token throughput from bucket counts without elapsed-time counters.

Version history

In vLLM v0.23.0, under the PrometheusStatLogger publisher with V1 metrics, vllm:iteration_tokens_total is registered as a Prometheus histogram carrying the label names model_name and engine.
Its v0.23.0 registration declares 13 literal histogram bucket boundaries spanning 1.0 to 16384.0.
In v0.23.0 the metric observes one histogram sample per engine step equal to computed prompt tokens plus generated tokens.
That v0.23.0 sampling is driven by iteration-level work, with the metric populated after each IterationStats update.
Despite the word total in its declared name, in v0.23.0 it describes the distribution of batch/step work rather than acting as a cumulative token counter.
Accordingly, in v0.23.0 the recommended use is to inspect batching through its distribution and not to calculate token throughput from bucket counts without elapsed-time counters.

Identity

Prometheus type: histogram
Labels: model_nameengine
Buckets: 13 boundaries · 1 → 16384
Unit: Not recorded — vLLM registration code has no structured unit field, and units are not inferred from name suffixes

Semantic coverage

covered · Availability & namecovered · Structural identitycovered · Measurement semanticscovered · Causal/runtime semanticscovered · Interpretation & compositioncovered · Guarded guidance

Metric knowledge (7)

Claim	Semantic layer	Verified
In vLLM v0.23.0, `vllm:iteration_tokens_total` observes one histogram sample per engine step equal to computed prompt tokens plus generated tokens.	Measurement semantics	Sep 29
In vLLM v0.23.0, `vllm:iteration_tokens_total` is populated from iteration-level work after each `IterationStats` update.	Causal/runtime semantics	Sep 29
In vLLM v0.23.0, `vllm:iteration_tokens_total` describes batch/step work distribution rather than a cumulative token counter despite total appearing in its declared name.	Interpretation & composition	Sep 29
In vLLM v0.23.0, `vllm:iteration_tokens_total` use its distribution to inspect batching; do not calculate token throughput from bucket counts without elapsed-time counters.	Guarded guidance	Sep 29
In vLLM v0.23.0, `vllm:iteration_tokens_total` is registered as a Prometheus histogram.	Structural identity	Sep 29
In vLLM v0.23.0, `vllm:iteration_tokens_total` is registered with the label names `model_name`, engine.	Structural identity	Sep 29
In vLLM v0.23.0, `vllm:iteration_tokens_total` declares 13 literal histogram bucket boundaries in its registration, from 1.0 to 16384.0.	Structural identity	Sep 29

Related Use Cases (1)

#35048: Aggregate throughput can hide experience regressions

Availability

vllm:iteration_tokens_total is registered as a Prometheus histogram in vLLM v0.23.0.

vllm.surface.prometheus_metricscomplete_inventoryPrometheusStatLogger metrics publisher

vLLM v0.23.0 v1 metrics loggers.py · vllm/v1/metrics/loggers.py:711-719