Prefix cache effectiveness metrics

seed slicevLLM v0.23.0evidence: source code and documentation

Coverage

Coverage for the vLLM v0.23.0 first-party prefix-cache counters and how to compute and interpret the prefix cache hit rate.

Strong for what vllm:prefix_cache_queries and vllm:prefix_cache_hits count (tokens), where they are set from, and how to compute hit rate; none for healthy hit-rate thresholds or workload-specific tuning.

Scoped to the two first-party prefix-cache counters; external/KV-connector prefix-cache counters are inventoried but not semantically covered.

Claims in this topic (5)

Claim	Verified
In vLLM v0.23.0, `vllm:prefix_cache_queries` is a Prometheus counter measured in queried tokens — it counts tokens, not requests or cache blocks.	4 hr ago
In vLLM v0.23.0, `vllm:prefix_cache_hits` is a Prometheus counter measured in cached (hit) tokens — it counts tokens, not requests or cache blocks.	4 hr ago
In vLLM v0.23.0, `vllm:prefix_cache_queries` is incremented from `SchedulerStats.prefix_cache_stats.queries` on each logging update and observes prefix-cache token reuse for the first-party prefix cache (KV-connector prefix-cache reuse is reported by the separate `vllm:external_prefix_cache_*` counters).	4 hr ago
In vLLM v0.23.0, `vllm:prefix_cache_hits` is incremented from `SchedulerStats.prefix_cache_stats.hits` on each logging update and observes prefix-cache token reuse for the first-party prefix cache (KV-connector prefix-cache reuse is reported by the separate `vllm:external_prefix_cache_*` counters).	4 hr ago
The vLLM v0.23.0 metrics design docs state the metric of interest is the prefix cache hit rate (hits per query): the counters are exposed raw so operators compute the rate over an interval of their choosing with PromQL (`rate(hits)/rate(queries)`); vLLM's own logging equivalent aggregates `hit_rate` over the most recent queries.	4 hr ago

Metrics in this topic (2)

prefix_cache_hits — 3 claims
prefix_cache_queries — 2 claims

Known gaps

The corpus has no benchmark or operator evidence for healthy prefix-cache hit-rate ranges; hit rate is workload-dependent (prompt-prefix sharing).

Schema.ai can explain how to compute the hit rate but must not grade an observed value as good or bad.

Check: Add benchmark or design-partner telemetry evidence before serving hit-rate thresholds.

vllm:external_prefix_cache_queries/hits (KV-connector prefix cache) are availability-inventoried but carry no semantic claims.

Existence answers are available for the external counters; interpretation claims are not.

Check: Promote KV-connector prefix-cache semantics from the connector metrics source when connector deployments become an eval target.