glossary/memory/cross-layer-kv-sharing
Cross-layer KV sharing
also cross-layer attention, layer-wise KV reuse, CSA2
Letting several transformer layers attend over one set of keys and values instead of each keeping its own, so the cache shrinks along the layer dimension the way grouped-query attention shrinks it along the head dimension. A layer in reuse mode inherits the previous layer's cache and its sparse selection outright; one in reindex mode inherits the cache but scores its own selection from it. DeepSeek-V4.1-Flash's CSA2 combines this with an encoder-decoder split, where the decoder's global KV is projected from the encoder state, and with 4-bit main KV, which is how it gets a million tokens of context into under a kilobyte per token.
890 B
Of global KV cache per token in HBM for DeepSeek-V4.1-Flash at 1M context, a quarter of V4-Flash's footprint, an eighth once the sliding-window cache is replayed instead of stored.
Related
- glossary/
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- KV cache quantization
Storing the cached keys and values in 8 or 4 bits instead of 16.
- Prefix caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.
- Cross-model KV cache transfer
Reusing the KV cache a small model built for a prompt as the starting cache of a larger model in the same family, so a cascade or router can escalate a request without the large model prefilling the context again.
- sources/
- topics/