vllm:request_queue_time_seconds

Summary

In vLLM v0.23.0, with V1 metrics published through PrometheusStatLogger, vllm:request_queue_time_seconds is registered as a Prometheus histogram. Its registration declares the label names model_name and engine. The registration also declares 21 literal histogram bucket boundaries, running from 0.3 to 7680.0. Each observation records one finished-request duration measured from queued_ts to the first scheduled_ts. In the scheduler's V1 finished-request accounting (IterationStats.update_from_finished_request), queued_time is computed as req_stats.scheduled_ts - req_stats.queued_ts, and the source comment describes this queued interval as spanning from the first QUEUED event to the first SCHEDULED. In the scheduler's V1 engine-core event accounting (IterationStats.update_from_events), each EngineCoreEventType.QUEUED event processed sets req_stats.queued_ts to event.timestamp. The metric is emitted on completion from scheduler lifecycle timestamps, so repeated QUEUED events overwrite queued_ts while the first SCHEDULED timestamp is retained. Interpretively, it measures the recorded waiting interval and includes only requests that reach finished-request accounting. For operational use, read its quantiles alongside running/KV/token-progress metrics to distinguish admission delay from decode delay; no universal threshold is established.

Version history

• In vLLM v0.23.0, the only version in scope here, vllm:request_queue_time_seconds is registered as a Prometheus histogram carrying the label names model_name and engine.
• Its v0.23.0 registration declares 21 literal histogram bucket boundaries spanning 0.3 to 7680.0.
• In v0.23.0 the histogram observes one duration per finished request, measured from queued_ts to the first scheduled_ts.
• That timestamp is populated in v0.23.0 by IterationStats.update_from_events, which sets req_stats.queued_ts to event.timestamp for each EngineCoreEventType.QUEUED event it processes.
• At finished-request accounting in v0.23.0, queued_time is computed as req_stats.scheduled_ts - req_stats.queued_ts, with the source comment describing the queued interval as running from the first QUEUED event to the first SCHEDULED.
• Emission in v0.23.0 happens on completion from scheduler lifecycle timestamps, where repeated QUEUED events overwrite queued_ts while the first SCHEDULED timestamp is retained.
• Interpreting the v0.23.0 metric, it reflects the recorded waiting interval and covers only requests that reach finished-request accounting.
• Guidance for v0.23.0 is to read its quantiles alongside running/KV/token-progress metrics to separate admission delay from decode delay, since no universal threshold is established.

Identity

Prometheus type

histogram

Labels

model_nameengine

Buckets

21 boundaries · 0.3 → 7680

Unit

Not recorded — vLLM registration code has no structured unit field, and units are not inferred from name suffixes

Semantic coverage

Metric knowledge (9)

Related Use Cases (1)

• #35048: Aggregate throughput can hide experience regressions

Availability