vLLM operational knowledgeSCHEMA.AI

vLLM (official site) is an open-source inference and serving engine for large language models. Schema.ai curates operational knowledge for running it: source-grounded, version-scoped, evidence-backed building blocks your AI tools can use to reason more confidently about your production deployment.

vLLM official sources

Source code, docs, design notes, issues — pinned and checked upstream material.

Schema.ai organizes

Deterministic, factual building blocks — version-accurate, verified, and concept-linked in a knowledge graph.

Your AI agent queries

Agents and assistants get facts that are always grounded in the original sources.

Try querying the Schema.ai API for vLLM knowledge — the same search an AI agent runs.

Search results for "Prefix cache effectiveness metrics"

Answerability

answerable_from_corpus

The local vLLM corpus has source-grounded blocks matching the query terms.

Matched topic

Prefix cache effectiveness metrics

Coverage for the vLLM v0.23.0 first-party prefix-cache counters and how to compute and interpret the prefix cache hit rate.

seed slice5 claimssource code and documentationfreshness: fresh

Answerability

Strong for what vllm:prefix_cache_queries and vllm:prefix_cache_hits count (tokens), where they are set from, and how to compute hit rate; none for healthy hit-rate thresholds or workload-specific tuning.

Known gaps

The corpus has no benchmark or operator evidence for healthy prefix-cache hit-rate ranges; hit rate is workload-dependent (prompt-prefix sharing).
vllm:external_prefix_cache_queries/hits (KV-connector prefix cache) are availability-inventoried but carry no semantic claims.

6 matching claims

Each result is a source-grounded claim — click through for its evidence, guards, and graph context. Ranked by: Lexical overlap against claim text, object values, applicability, and entity names.

1Claimcaveatconfidence: highfreshness: fresh · full

In vLLM v0.23.0, `vllm:prefix_cache_queries` is incremented from `SchedulerStats.prefix_cache_stats.queries` on each logging update and observes prefix-cache token reuse for the first-party prefix cache (KV-connector prefix-cache reuse is reported by the separate `vllm:external_prefix_cache_*` counters).

Applies when

vllmv0.23.0PrometheusStatLogger metrics publisherV1 metricsbasis: vLLM v0.23.0 source-code metric registration, scheduler stats, and metrics design documentation

Topics

Prefix cache effectiveness metrics

Sources

source.vllm.code.v1_metrics_loggers.v0230source.vllm.code.v1_metrics_stats.v0230source.vllm.code.v1_metrics_loggers.v0230

Does not establish (2)

why reuse is high or low for a given workload
connector/external prefix-cache reuse

Matched query terms against entity.vllm.metric.prefix_cache_queries and v0.23.0.

View claim →

2Claimcaveatconfidence: highfreshness: fresh · full

In vLLM v0.23.0, `vllm:prefix_cache_hits` is incremented from `SchedulerStats.prefix_cache_stats.hits` on each logging update and observes prefix-cache token reuse for the first-party prefix cache (KV-connector prefix-cache reuse is reported by the separate `vllm:external_prefix_cache_*` counters).

Applies when

vllmv0.23.0PrometheusStatLogger metrics publisherV1 metricsbasis: vLLM v0.23.0 source-code metric registration, scheduler stats, and metrics design documentation

Topics

Prefix cache effectiveness metrics

Sources

source.vllm.code.v1_metrics_loggers.v0230source.vllm.code.v1_metrics_stats.v0230source.vllm.code.v1_metrics_loggers.v0230

Does not establish (2)

why reuse is high or low for a given workload
connector/external prefix-cache reuse

Matched query terms against entity.vllm.metric.prefix_cache_hits and v0.23.0.

View claim →

3Claimcaveatconfidence: highfreshness: fresh · full

The vLLM v0.23.0 metrics design docs state the metric of interest is the prefix cache hit rate (hits per query): the counters are exposed raw so operators compute the rate over an interval of their choosing with PromQL (`rate(hits)/rate(queries)`); vLLM's own logging equivalent aggregates `hit_rate` over the most recent queries.

Applies when

vllmv0.23.0PrometheusStatLogger metrics publisherV1 metricsbasis: vLLM v0.23.0 source-code metric registration, scheduler stats, and metrics design documentation

Topics

Prefix cache effectiveness metrics

Sources

source.vllm.docs.design_metrics.v0230

Does not establish (2)

a healthy or target hit rate (workload-dependent)
how to increase hit rate for a given workload

Matched query terms against entity.vllm.metric.prefix_cache_hits and v0.23.0.

View claim →

4Claimcaveatconfidence: highfreshness: fresh · full

In vLLM v0.23.0, `vllm:prefix_cache_queries` is a Prometheus counter measured in queried tokens — it counts tokens, not requests or cache blocks.

Applies when

vllmv0.23.0PrometheusStatLogger metrics publisherV1 metricsbasis: vLLM v0.23.0 source-code metric registration, scheduler stats, and metrics design documentation

Topics

Prefix cache effectiveness metrics

Sources

source.vllm.code.v1_metrics_loggers.v0230source.vllm.code.v1_metrics_loggers.v0230

Does not establish (3)

a healthy hit-rate threshold
request-level cache behavior (the unit is tokens, not requests)
cache sizing guidance

Matched query terms against entity.vllm.metric.prefix_cache_queries and v0.23.0.

View claim →

5Claimcaveatconfidence: highfreshness: fresh · full

In vLLM v0.23.0, `vllm:prefix_cache_hits` is a Prometheus counter measured in cached (hit) tokens — it counts tokens, not requests or cache blocks.

Applies when

vllmv0.23.0PrometheusStatLogger metrics publisherV1 metricsbasis: vLLM v0.23.0 source-code metric registration, scheduler stats, and metrics design documentation

Topics

Prefix cache effectiveness metrics

Sources

source.vllm.code.v1_metrics_loggers.v0230source.vllm.code.v1_metrics_loggers.v0230

Does not establish (3)

a healthy hit-rate threshold
request-level cache behavior (the unit is tokens, not requests)
cache sizing guidance

Matched query terms against entity.vllm.metric.prefix_cache_hits and v0.23.0.

View claim →

6Claimcaveatconfidence: highfreshness: fresh · full

In vLLM v0.23.0, `vllm:kv_cache_usage_perc` is a Prometheus gauge for KV-cache usage where 1 means 100 percent usage.

Applies when

vllmv0.23.0PrometheusStatLogger metrics publisherV1 metricsbasis: vLLM v0.23.0 source-code metric registration and metrics design documentation

Topics

Scheduler and KV pressure metrics

Sources

source.vllm.code.v1_metrics_loggers.v0230

Does not establish (3)

how much free KV cache remains for a specific model
whether a workload will preempt
a universal alert threshold

Matched query terms against entity.vllm.metric.kv_cache_usage_perc and v0.23.0.

View claim →

Provenance & sources

Scope: local_corpus_snapshots

vLLM v0.23.0 v1 metrics loggers.py

vllm-projectsource_filev0.23.0

vLLM v0.23.0 v1 metrics stats.py

vllm-projectsource_filev0.23.0

vLLM v0.23.0 metrics design docs

vllm-projectdocs_pagev0.23.0

How to use this

Use as

Source-grounded building blocks for a downstream UI, evaluator, or human.

Do not use as

A complete recommendation or comprehensive vLLM coverage map.

Next steps

Inspect returned blocks and their source snapshots. — Schema returns evidence-backed ingredients rather than a composed answer.

Search results for "Prefix cache effectiveness metrics"

Answerability

Matched topic

6 matching claims

In vLLM v0.23.0, vllm:prefix_cache_queries is incremented from SchedulerStats.prefix_cache_stats.queries on each logging update and observes prefix-cache token reuse for the first-party prefix cache (KV-connector prefix-cache reuse is reported by the separate vllm:external_prefix_cache_* counters).

In vLLM v0.23.0, vllm:prefix_cache_hits is incremented from SchedulerStats.prefix_cache_stats.hits on each logging update and observes prefix-cache token reuse for the first-party prefix cache (KV-connector prefix-cache reuse is reported by the separate vllm:external_prefix_cache_* counters).

In vLLM v0.23.0, vllm:prefix_cache_queries is a Prometheus counter measured in queried tokens — it counts tokens, not requests or cache blocks.

In vLLM v0.23.0, vllm:prefix_cache_hits is a Prometheus counter measured in cached (hit) tokens — it counts tokens, not requests or cache blocks.

In vLLM v0.23.0, vllm:kv_cache_usage_perc is a Prometheus gauge for KV-cache usage where 1 means 100 percent usage.

Provenance & sources

How to use this

In vLLM v0.23.0, `vllm:prefix_cache_hits` is incremented from `SchedulerStats.prefix_cache_stats.hits` on each logging update and observes prefix-cache token reuse for the first-party prefix cache (KV-connector prefix-cache reuse is reported by the separate `vllm:external_prefix_cache_*` counters).

In vLLM v0.23.0, `vllm:prefix_cache_queries` is a Prometheus counter measured in queried tokens — it counts tokens, not requests or cache blocks.

In vLLM v0.23.0, `vllm:prefix_cache_hits` is a Prometheus counter measured in cached (hit) tokens — it counts tokens, not requests or cache blocks.

In vLLM v0.23.0, `vllm:kv_cache_usage_perc` is a Prometheus gauge for KV-cache usage where 1 means 100 percent usage.