glossary/memory/paged-attention
PagedAttention
Storing the KV cache in fixed-size blocks that need not be contiguous, with a per-request table mapping its logical positions to physical blocks, exactly as an operating system pages virtual memory. Without it a request reserves its maximum possible length up front and most of that sits empty. With it, memory is handed out a block at a time as tokens arrive, and a block can be shared by every request whose prefix matches.
20 to 40%
The share of reserved KV memory actually holding tokens under contiguous allocation, per the vLLM paper, and reproduced in the explainer.
See it happen
Related
- glossary/
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- Block table
The per-request list that says which physical block of KV cache holds each run of its tokens.
- Prefix caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.
- Preemption
Evicting a running request from the batch when the KV cache runs out, so the others can keep growing.
- sources/
- topics/