feed/tags/memory
memory
3 explainers you can run, 8 glossary terms, and one reading ordered as a path, each with one line on why it earns your time.
- beginnerThe KV cache is the thing you are actually renting
Fill a GPU with weights and per-request cache until it will not take another user. 320 KiB per token adds up faster than anyone expects.
- beginnerReserve for the worst case, or page
Forty requests of unknown length against one memory, under two allocators at once. The PagedAttention paper's 20 to 40% utilisation figure, reproduced rather than quoted.
- intermediateThe hit rate is the workload
Agent sessions taking turns against one card's memory. Add sessions and watch each turn's cost jump from nothing to ten seconds as its history is evicted.
- Preemption
Evicting a running request from the batch when the KV cache runs out, so the others can keep growing.
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- PagedAttention
Storing the KV cache in fixed-size blocks that need not be contiguous, with a per-request table mapping its logical positions to physical blocks, exactly as an operating system pages virtual memory.
- Block table
The per-request list that says which physical block of KV cache holds each run of its tokens.
- Cache eviction
Dropping stored KV cache to make room for new requests.
- High bandwidth memory
The stacked DRAM on the GPU package that holds the weights and the KV cache.
- KV cache offloading
Spilling evicted KV cache to host DRAM, or further to SSD or a remote store, instead of throwing it away.
- Working set
The amount of KV cache the active sessions need resident to keep hitting.
Current work, at the depth practitioners actually argue about.