glossary/memory/kv-cache
KV cache
also key-value cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing. It grows with every token of every request in flight, which makes it the memory that scales with users while the weights stay fixed. It is what caps batch size, what paging manages, what a prefix cache shares, and what disaggregation ships between machines.
320 KiB
Per token for Llama 3 70B in BF16: 80 layers, 8 KV heads, 128 dimensions, keys and values, 2 bytes each.
See it happen
Related
- glossary/
- PagedAttention
Storing the KV cache in fixed-size blocks that need not be contiguous, with a per-request table mapping its logical positions to physical blocks, exactly as an operating system pages virtual memory.
- Prefix caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.
- High bandwidth memory
The stacked DRAM on the GPU package that holds the weights and the KV cache.
- KV cache quantization
Storing the cached keys and values in 8 or 4 bits instead of 16.
- sources/
- topics/