glossary/memory/prefix-cache-hit-rate
Prefix cache hit rate
also cache hit rate, KV cache hit rate
The share of prompt tokens found already in the cache rather than prefilled. It is a property of the workload against the memory, not of the cache code: a few sessions taking turns against one GPU hit nearly always, and enough sessions that their histories no longer fit evict each other in turn and hit nearly never. For agent workloads with long contexts it is the single number that decides cost per turn.
5 sessions
Of 88K-token agent history that fit in 180 GB of cache across four A100s before eviction begins, once each has grown over eight turns.
See it happen
Related
- glossary/
- Prefix caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.
- Cache eviction
Dropping stored KV cache to make room for new requests.
- Working set
The amount of KV cache the active sessions need resident to keep hitting.
- KV cache offloading
Spilling evicted KV cache to host DRAM, or further to SSD or a remote store, instead of throwing it away.
- sources/
- topics/