glossary/memory/prefix-caching
Prefix caching
also automatic prefix caching, prompt caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it. Attention over a prefix depends only on the prefix, so the cache is identical for anyone who sends those tokens. System prompts shared by every user and the earlier turns of a conversation are the two big wins. The first token then costs a cache lookup instead of a prefill.
80%
Of prefill compute saved after eleven requests in the explainer: eight users, two system prompts of 2K tokens, three turns each.
See it happen
Related
- glossary/
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- RadixAttention
SGLang's prefix cache: every stored sequence is a path in a radix tree keyed by tokens, so a new request walks the tree as far as its prompt matches and computes only the remainder.
- Cache eviction
Dropping stored KV cache to make room for new requests.
- Prefix cache hit rate
The share of prompt tokens found already in the cache rather than prefilled.
- sources/
- SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al., 2023)
- vLLM: Automatic Prefix Caching
- topics/