Schema.ai· vLLM
vLLM official site Log in
vLLM v0.23.0 · Version scoped

vllm:inter_token_latency_seconds

Histogram of inter-token latency in seconds.

MetricPrometheusverified Sep 29

Summary

In vLLM v0.23.0, under the PrometheusStatLogger metrics publisher with V1 metrics, vllm:inter_token_latency_seconds is registered as a Prometheus histogram. Its registration in that version declares 19 literal histogram bucket boundaries, ranging from 0.01 to 80.0. It is registered with the label names model_name and engine. Under PrometheusStatLogger with V1 metrics in v0.23.0, each observation is an interval in seconds between consecutive non-prefill token timestamps. In that same setup, one observation is appended for every decode update that follows the first-token/prefill update. Consequently, the metric is token-interval weighted and differs from the one-sample-per-request TPOT histogram. Use it to inspect token-level decode cadence, and do not equate its histogram count with requests or its mean with request-weighted TPOT.

Version history

  • The only version on record for this metric is vLLM v0.23.0, where vllm:inter_token_latency_seconds is registered as a Prometheus histogram with the label names model_name and engine under the PrometheusStatLogger V1 metrics publisher.
  • In that same v0.23.0 registration, the histogram declares 19 literal bucket boundaries spanning 0.01 to 80.0 seconds.
  • As of v0.23.0, each observation records an interval in seconds between consecutive non-prefill token timestamps.
  • At runtime in v0.23.0, one observation is appended for every decode update that follows the first-token/prefill update.
  • Because of that accumulation, the v0.23.0 metric is token-interval weighted and therefore differs from the one-sample-per-request TPOT histogram.
  • Accordingly, in v0.23.0 it should be used to inspect token-level decode cadence, without equating its histogram count with requests or its mean with request-weighted TPOT.

Identity

Prometheus type
histogram
Labels
model_nameengine
Buckets
19 boundaries · 0.01 → 80
Unit
Not recorded — vLLM registration code has no structured unit field, and units are not inferred from name suffixes

Semantic coverage

covered · Availability & namecovered · Structural identitycovered · Measurement semanticscovered · Causal/runtime semanticscovered · Interpretation & compositioncovered · Guarded guidance

Metric knowledge (7)

ClaimSemantic layerVerified
In vLLM v0.23.0, vllm:inter_token_latency_seconds observes an interval in seconds between consecutive non-prefill token timestamps.Measurement semanticsSep 29
In vLLM v0.23.0, vllm:inter_token_latency_seconds appends one observation for each decode update after the first-token/prefill update.Causal/runtime semanticsSep 29
In vLLM v0.23.0, vllm:inter_token_latency_seconds is token-interval weighted and differs from the one-sample-per-request TPOT histogram.Interpretation & compositionSep 29
In vLLM v0.23.0, vllm:inter_token_latency_seconds use it to inspect token-level decode cadence; do not equate its histogram count with requests or its mean with request-weighted TPOT.Guarded guidanceSep 29
In vLLM v0.23.0, vllm:inter_token_latency_seconds is registered as a Prometheus histogram.Structural identitySep 29
In vLLM v0.23.0, vllm:inter_token_latency_seconds is registered with the label names model_name, engine.Structural identitySep 29
In vLLM v0.23.0, vllm:inter_token_latency_seconds declares 19 literal histogram bucket boundaries in its registration, from 0.01 to 80.0.Structural identitySep 29

Related Use Cases (1)

Availability

vllm:inter_token_latency_seconds is registered as a Prometheus histogram in vLLM v0.23.0.
vllm.surface.prometheus_metricscomplete_inventoryPrometheusStatLogger metrics publisher

vLLM v0.23.0 v1 metrics loggers.py · vllm/v1/metrics/loggers.py:787-815