feed/tags/kv-cache
KV cache
4 explainers you can run, 9 glossary terms, and 5 readings ordered as a path, each with one line on why it earns your time.
- beginnerThe KV cache is the thing you are actually renting
Fill a GPU with weights and per-request cache until it will not take another user. 320 KiB per token adds up faster than anyone expects.
- beginnerReserve for the worst case, or page
Forty requests of unknown length against one memory, under two allocators at once. The PagedAttention paper's 20 to 40% utilisation figure, reproduced rather than quoted.
- beginnerCompute the system prompt once
Eight users, one system prompt, several turns each. Step through the requests and watch what each one actually computes once the cache is addressed by prefix.
- intermediateThe hit rate is the workload
Agent sessions taking turns against one card's memory. Add sessions and watch each turn's cost jump from nothing to ten seconds as its history is evicted.
- Preemption
Evicting a running request from the batch when the KV cache runs out, so the others can keep growing.
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- PagedAttention
Storing the KV cache in fixed-size blocks that need not be contiguous, with a per-request table mapping its logical positions to physical blocks, exactly as an operating system pages virtual memory.
- Block table
The per-request list that says which physical block of KV cache holds each run of its tokens.
- Prefix caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.
- RadixAttention
SGLang's prefix cache: every stored sequence is a path in a radix tree keyed by tokens, so a new request walks the tree as far as its prompt matches and computes only the remainder.
- Cache eviction
Dropping stored KV cache to make room for new requests.
- KV cache offloading
Spilling evicted KV cache to host DRAM, or further to SSD or a remote store, instead of throwing it away.
- KV cache quantization
Storing the cached keys and values in 8 or 4 bits instead of 16.
Start here. No serving experience assumed.
The mental models the rest of the field takes for granted.
Current work, at the depth practitioners actually argue about.
- toolNIXL: NVIDIA Inference Xfer Library
Disaggregation only pays if the KV cache moves between pools fast enough. This is the transfer layer underneath Dynamo doing that job.
#disaggregation#kv-cache#networking - articleAgentX and InferenceX v3: does the CUDA moat hold up in agentic inference?
An agent turn is 88K tokens in and 413 out, so the benchmark is really a KV cache hit-rate benchmark: 91% on B300 at 384 sessions, 73% on B200 with the rest spilling to DRAM. Nvidia, AMD, five open models, 70 upstream PRs. Partly paywalled.
Cam Quilici, Bryan Shan, Alec Ibarra and others#agents#prefix-cache#kv-cache#benchmark#hardware - articleThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale
An agent that pauses for a tool call gets its cache evicted and pays to rebuild it. Scheduling the whole program instead of each request is the fix.
Hao Kang and colleagues#agents#kv-cache#scheduling