Schema.ai#12077: Active KV use is not reusable cache residency
Zero active use does not mean cached prefixes were erased
Reported environment
OPT-6.7B; automatic prefix caching; swap preemption; ShareGPT-derived burst workload.
The nuance
The usage gauge measures active/referenced capacity, while released blocks may remain reusable until eviction.
Expected behavior
Explain active use versus reusable residency and use hit/query counters for effectiveness; never infer cache loss from the gauge alone.
What Schema establishes
Semantics resolve the interpretation; actual reuse requires same-window hit/query observations.
Runtime evidence still needed
- Same-label same-window prefix hit/query rates
- Controlled repeated-prefix request before eviction
Try the questions (2)
Why does KV cache usage fall after requests finish even though prefix caching is enabled?run →Does zero vllm:kv_cache_usage_perc mean reusable prefixes were erased?run →Grounding claims
- In vLLM v0.23.0,
vllm:kv_cache_usage_percis set fromSchedulerStats.kv_cache_usageand observes KV-cache usage pressure. - In vLLM v0.23.0,
vllm:prefix_cache_queriesis incremented fromSchedulerStats.prefix_cache_stats.querieson each logging update and observes prefix-cache token reuse for the first-party prefix cache (KV-connector prefix-cache reuse is reported by the separatevllm:external_prefix_cache_*counters). - In vLLM v0.23.0,
vllm:prefix_cache_hitsis incremented fromSchedulerStats.prefix_cache_stats.hitson each logging update and observes prefix-cache token reuse for the first-party prefix cache (KV-connector prefix-cache reuse is reported by the separatevllm:external_prefix_cache_*counters). - The vLLM v0.23.0 metrics design docs state the metric of interest is the prefix cache hit rate (hits per query): the counters are exposed raw so operators compute the rate over an interval of their choosing with PromQL (
rate(hits)/rate(queries)); vLLM's own logging equivalent aggregateshit_rateover the most recent queries.