glossary/memory/cache-eviction
Cache eviction
also LRU eviction
Dropping stored KV cache to make room for new requests. Least-recently-used is the usual rule: the prefix nobody has touched for longest goes first. What is evicted is either gone, and costs a full prefill to rebuild, or spilled to host memory, and costs a copy back over PCIe. Eviction is the mechanism behind a falling hit rate; the policy only decides whose turn is expensive.
10.1 s
To recompute an evicted 88K-token context for Llama 3 70B on four A100s, versus 2.5 s to pull it back from host DRAM.
See it happen
Related
- glossary/
- Prefix cache hit rate
The share of prompt tokens found already in the cache rather than prefilled.
- KV cache offloading
Spilling evicted KV cache to host DRAM, or further to SSD or a remote store, instead of throwing it away.
- Working set
The amount of KV cache the active sessions need resident to keep hitting.
- Preemption
Evicting a running request from the batch when the KV cache runs out, so the others can keep growing.
- sources/
- topics/