vllm:inter_token_latency_seconds

Histogram of inter-token latency in seconds.

MetricPrometheusvLLM v0.23.0verified 10 hr ago

Identity

Prometheus type: histogram
Labels: model_nameengine
Buckets: 19 boundaries · 0.01 → 80
Unit: Not recorded — vLLM registration code has no structured unit field, and units are not inferred from name suffixes

Semantic coverage

covered · Availability & namecovered · Structural identitycovered · Measurement semanticscovered · Causal/runtime semanticscovered · Interpretation & compositioncovered · Guarded guidance

Metric knowledge (7)

Claim	Semantic layer	Verified
In vLLM v0.23.0, `vllm:inter_token_latency_seconds` observes an interval in seconds between consecutive non-prefill token timestamps.	Measurement semantics	10 hr ago
In vLLM v0.23.0, `vllm:inter_token_latency_seconds` appends one observation for each decode update after the first-token/prefill update.	Causal/runtime semantics	10 hr ago
In vLLM v0.23.0, `vllm:inter_token_latency_seconds` is token-interval weighted and differs from the one-sample-per-request `TPOT` histogram.	Interpretation & composition	10 hr ago
In vLLM v0.23.0, `vllm:inter_token_latency_seconds` use it to inspect token-level decode cadence; do not equate its histogram count with requests or its mean with request-weighted `TPOT`.	Guarded guidance	10 hr ago
In vLLM v0.23.0, `vllm:inter_token_latency_seconds` is registered as a Prometheus histogram.	Structural identity	10 hr ago
In vLLM v0.23.0, `vllm:inter_token_latency_seconds` is registered with the label names `model_name`, engine.	Structural identity	10 hr ago
In vLLM v0.23.0, `vllm:inter_token_latency_seconds` declares 19 literal histogram bucket boundaries in its registration, from 0.01 to 80.0.	Structural identity	10 hr ago

Related Use Cases (1)

#35048: Aggregate throughput can hide experience regressions

Availability

vllm:inter_token_latency_seconds is registered as a Prometheus histogram in vLLM v0.23.0.

vllm.surface.prometheus_metricscomplete_inventoryPrometheusStatLogger metrics publisher

vLLM v0.23.0 v1 metrics loggers.py · vllm/v1/metrics/loggers.py:787-815