Schema.ai· vLLM· Schema 0.2.0
vLLM official site vLLM v0.23.0Log in

Metrics

NameDescriptionTopicsProviderVersionVerifiedSources
num_requests_runningGauge for the number of requests in model execution batches.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.pymetrics/stats.py
num_requests_waitingGauge for the number of requests waiting to be processed.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
num_requests_waiting_by_reasonGauge that breaks waiting requests into capacity and deferred reasons.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
kv_cache_usage_percGauge for KV-cache usage where 1 means 100 percent usage.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.pymetrics/stats.py
num_preemptionsCounter for cumulative preemptions from the engine.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.pymetrics/stats.py
engine_sleep_stateGauge for engine sleep state; sleep_state label values awake, weights_offloaded, and discard_all encode sleep levels.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
corrupted_requestsCounter for corrupted requests, measured as requests with NaNs in logits.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
prefix_cache_queriesCounter for prefix cache queries, in queried tokens.Prefix cache effectiveness metrics, Scheduler and KV pressure metrics +1Prometheusv0.23.07 hr agometrics/loggers.pymetrics/stats.py
prefix_cache_hitsCounter for prefix cache hits, in cached tokens.Prefix cache effectiveness metrics, Scheduler and KV pressure metrics +1Prometheusv0.23.07 hr agometrics/loggers.pymetrics/stats.pydesign/metrics.md
external_prefix_cache_queriesCounter for external prefix cache queries from KV connector cross-instance cache sharing, in queried tokens.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
external_prefix_cache_hitsCounter for external prefix cache hits from KV connector cross-instance cache sharing, in cached tokens.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
mm_cache_queriesCounter for multi-modal cache queries, in queried items.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
mm_cache_hitsCounter for multi-modal cache hits, in cached items.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
prompt_tokensCounter for the number of prefill tokens processed.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
prompt_tokens_by_sourceCounter for prompt tokens broken down by a source label.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
prompt_tokens_cachedCounter for cached prompt tokens, local plus external.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
generation_tokensCounter for the number of generation tokens processed.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
request_successCounter for successfully processed requests, labeled by finished_reason.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
request_prompt_tokensHistogram of prefill token counts per request.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
request_generation_tokensHistogram of generation token counts per request.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
iteration_tokens_totalHistogram of the number of tokens per engine step.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
request_max_num_generation_tokensHistogram of the maximum requested generation tokens per request.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
request_params_nHistogram of the n request parameter.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
request_params_max_tokensHistogram of the max_tokens request parameter.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
time_to_first_token_secondsHistogram of time to first token in seconds.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
inter_token_latency_secondsHistogram of inter-token latency in seconds.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
request_time_per_output_token_secondsHistogram of mean time per output token per request.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
e2e_request_latency_secondsHistogram of end-to-end request latency in seconds.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
request_queue_time_secondsHistogram of time spent in the WAITING phase per request.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.pymetrics/stats.py
request_inference_time_secondsHistogram of time spent in the RUNNING phase per request.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.pymetrics/stats.py
request_prefill_time_secondsHistogram of time spent in the PREFILL phase per request.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.pymetrics/stats.py
request_decode_time_secondsHistogram of time spent in the DECODE phase per request.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.pymetrics/stats.py
request_prefill_kv_computed_tokensHistogram of new KV tokens computed during prefill, excluding cached tokens.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
kv_block_lifetime_secondsHistogram of KV cache block lifetime from allocation to eviction; sampled metric.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
kv_block_idle_before_evict_secondsHistogram of idle time before KV cache block eviction; sampled metric.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
kv_block_reuse_gap_secondsHistogram of time gaps between consecutive KV cache block accesses; sampled metric.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
lora_requests_infoGauge with running stats on LoRA requests, labeled by max_lora and running/waiting adapter lists.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
cache_config_infoInfo-style gauge permanently set to 1 that carries LLMEngine CacheConfig fields as labels; registered at engine initialization.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/loggers.py
spec_decode_num_draftsCounter for the number of speculative decoding drafts.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agospec_decode/metrics.py
spec_decode_num_draft_tokensCounter for the number of speculative draft tokens.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agospec_decode/metrics.py
spec_decode_num_accepted_tokensCounter for the number of accepted speculative tokens.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agospec_decode/metrics.py
spec_decode_num_accepted_tokens_per_posCounter for accepted speculative tokens per draft position, labeled by position.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agospec_decode/metrics.py
estimated_flops_per_gpu_totalCounter of estimated floating point operations per GPU, for Model FLOPs Utilization calculations.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/perf.py
estimated_read_bytes_per_gpu_totalCounter of estimated bytes read from memory per GPU, for Model FLOPs Utilization calculations.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/perf.py
estimated_write_bytes_per_gpu_totalCounter of estimated bytes written to memory per GPU, for Model FLOPs Utilization calculations.Scheduler and KV pressure metrics, vLLM v0.23.0 metric identityPrometheusv0.23.07 hr agometrics/perf.py