glossary/memory/radix-attention
RadixAttention
also radix tree cache
SGLang's prefix cache: every stored sequence is a path in a radix tree keyed by tokens, so a new request walks the tree as far as its prompt matches and computes only the remainder. Shared prefixes are shared nodes, which is what lets many users' contexts hold one copy of a system prompt. Eviction is least-recently-used over leaves, so the cache keeps what is being extended and drops what has gone quiet.
one node
Per shared system prompt, however many users are attached below it.
See it happen
Related
- glossary/
- Prefix caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.
- Cache eviction
Dropping stored KV cache to make room for new requests.
- SGLang
A serving engine and frontend language built around RadixAttention, its radix-tree prefix cache, and a compressed finite-state machine for fast constrained decoding.
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- sources/
- topics/