glossary/memory/kv-cache-offloading

KV cache offloading

also KV cache tiering, CPU offload

Spilling evicted KV cache to host DRAM, or further to SSD or a remote store, instead of throwing it away. Host memory is far larger than HBM and far slower to reach, so a spilled context costs a copy over PCIe rather than a full prefill. Whether that is a win depends on the size: for long agent contexts the copy is a quarter of the recompute. The tiers form a hierarchy with recompute at the bottom.

See it happen

• explainers/
• intermediateThe hit rate is the workload

  Agent sessions taking turns against one card's memory. Add sessions and watch each turn's cost jump from nothing to ten seconds as its history is evicted.

Related

• glossary/
• Cache eviction

  Dropping stored KV cache to make room for new requests.

• High bandwidth memory

  The stacked DRAM on the GPU package that holds the weights and the KV cache.

• Prefix cache hit rate

  The share of prompt tokens found already in the cache rather than prefilled.

• Disaggregated serving

  Running prefill and decode on separate pools of GPUs and shipping the KV cache between them.

• sources/
• LMCache
• NIXL: NVIDIA Inference Xfer Library
• topics/
• #kv-cache#memory#networking