Schema.ai· vLLM· Schema 0.2.1
vLLM official site vLLM v0.23.0Log in

Metrics

NameDescriptionTopicsDepthUse casesProviderVersionVerifiedSources
num_requests_runningGauge for the number of requests in model execution batches.Scheduler and KV pressure metrics, vLLM metric identity6/61Prometheusv0.23.010 hr agometrics/loggers.pymetrics/stats.py
kv_cache_usage_percGauge for KV-cache usage where 1 means 100 percent usage.Scheduler and KV pressure metrics, vLLM metric identity6/61Prometheusv0.23.010 hr agometrics/loggers.pymetrics/stats.py
prefix_cache_queriesCounter for prefix cache queries, in queried tokens.Prefix cache effectiveness metrics, Scheduler and KV pressure metrics +16/63Prometheusv0.23.010 hr agometrics/loggers.pymetrics/stats.py
prefix_cache_hitsCounter for prefix cache hits, in cached tokens.Prefix cache effectiveness metrics, Scheduler and KV pressure metrics +16/63Prometheusv0.23.010 hr agometrics/loggers.pymetrics/stats.pydesign/metrics.md
prompt_tokensCounter for the number of prefill tokens processed.Scheduler and KV pressure metrics, vLLM metric identity6/61Prometheusv0.23.010 hr agometrics/loggers.py
prompt_tokens_cachedCounter for cached prompt tokens, local plus external.Scheduler and KV pressure metrics, vLLM metric identity6/61Prometheusv0.23.010 hr agometrics/loggers.py
generation_tokensCounter for the number of generation tokens processed.Scheduler and KV pressure metrics, vLLM metric identity6/61Prometheusv0.23.010 hr agometrics/loggers.py
request_successCounter for successfully processed requests, labeled by finished_reason.Scheduler and KV pressure metrics, vLLM metric identity6/62Prometheusv0.23.010 hr agometrics/loggers.py
iteration_tokens_totalHistogram of the number of tokens per engine step.Scheduler and KV pressure metrics, vLLM metric identity6/61Prometheusv0.23.010 hr agometrics/loggers.py
time_to_first_token_secondsHistogram of time to first token in seconds.Scheduler and KV pressure metrics, vLLM metric identity6/62Prometheusv0.23.010 hr agometrics/loggers.pymetrics/stats.py
inter_token_latency_secondsHistogram of inter-token latency in seconds.Scheduler and KV pressure metrics, vLLM metric identity6/61Prometheusv0.23.010 hr agometrics/loggers.pymetrics/stats.py
request_time_per_output_token_secondsHistogram of mean time per output token per request.Scheduler and KV pressure metrics, vLLM metric identity6/61Prometheusv0.23.010 hr agometrics/loggers.pymetrics/stats.py
e2e_request_latency_secondsHistogram of end-to-end request latency in seconds.Scheduler and KV pressure metrics, vLLM metric identity6/61Prometheusv0.23.010 hr agometrics/loggers.pymetrics/stats.py
request_queue_time_secondsHistogram of time spent in the WAITING phase per request.Scheduler and KV pressure metrics, vLLM metric identity6/61Prometheusv0.23.010 hr agometrics/loggers.pymetrics/stats.py
request_prefill_kv_computed_tokensHistogram of new KV tokens computed during prefill, excluding cached tokens.Scheduler and KV pressure metrics, vLLM metric identity6/61Prometheusv0.23.010 hr agometrics/loggers.py