Schema.ai#38194: Prefix-cache percentages require comparable definitions
Two hit rates are not comparable until their populations and scopes match
Reported environment
High-concurrency multi-turn workload comparing vLLM with a hosted API; no vLLM version reported.
The nuance
A percentage hides its numerator, denominator, labels, process scope, routing, reset history, tokenization, and time window.
Expected behavior
Give the vLLM token-rate formula and comparison preconditions; do not diagnose a regression from an undocumented percentage.
What Schema establishes
Schema defines the vLLM side; the other system and matched runtime data are required for comparison.
Runtime evidence still needed
- Both formulas and scopes
- Identical rendered token streams and routing
- Matched warm-up and windows
Try the questions (2)
Why is vLLM prefix cache hit rate lower than another providerrun →Can I compare vLLM and provider cache percentages directlyrun →Grounding claims
- In vLLM v0.23.0,
vllm:prefix_cache_queriesis a Prometheus counter measured in queried tokens — it counts tokens, not requests or cache blocks. - In vLLM v0.23.0,
vllm:prefix_cache_hitsis a Prometheus counter measured in cached (hit) tokens — it counts tokens, not requests or cache blocks. - The vLLM v0.23.0 metrics design docs state the metric of interest is the prefix cache hit rate (hits per query): the counters are exposed raw so operators compute the rate over an interval of their choosing with PromQL (
rate(hits)/rate(queries)); vLLM's own logging equivalent aggregateshit_rateover the most recent queries.