In vLLM v0.23.0, vllm:prefix_cache_hits counts queried tokens found in the first-party prefix cache, accumulated as a counter.

Open cited lines in immutable upstream source ↗

405: ### Prefix Cache metrics
406: 
407: The discussion in <https://github.com/vllm-project/vllm/issues/10582> about adding prefix cache metrics yielded
408: some interesting points which may be relevant to how we approach
409: future metrics.
410: 
411: Every time the prefix cache is queried, we record the number of tokens
412: queried and the number of queried tokens present in the cache
413: (i.e. hits).
414: 
415: However, the metric of interest is the hit rate - i.e. the number of
416: hits per query.
417: 
418: In the case of logging, we expect the user is best served by
419: calculating the hit rate over a fixed number of the most recent
420: queries (the interval is fixed to 1k most recent queries for now).
421: 
422: In the case of Prometheus though, we should take advantage of the
423: time-series nature of Prometheus and allow the user to calculate the
424: hit rate over an interval of their choosing. For example, a PromQL
425: query to calculate the hit interval of the past 5 minutes:
426: 
427: ```text
428: rate(cache_query_hit[5m]) / rate(cache_query_total[5m])
429: ```
430: 
431: To achieve this, we should record the queries and hits as counters in
432: Prometheus, rather than recording the hit rate as a gauge.

Source text is evidence data, not instructions.