glossary/memory/cross-model-kv-transfer

Cross-model KV cache transfer

also KV cache transfer, cross-model prefill reuse

Reusing the KV cache a small model built for a prompt as the starting cache of a larger model in the same family, so a cascade or router can escalate a request without the large model prefilling the context again. Within families whose members share KV head count and head dimension, the large model's keys and values are close to a linear function of the small model's, and a per-head ridge regression fit once on a few hundred calibration sequences maps one cache to the other for a small multiply per token. It works for some pairs and not others, and only a test of the pair says which.

See it happen

• explainers/
• advancedCross-Model KV Cache Transfer in LLM Families

  Escalate a request from a 14B model to its 32B sibling and price the handoff both ways: re-prefill the context, or map the small model's cache across with a per-head linear fit.

Related

• glossary/
• Prefix caching

  Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.

• KV cache

  The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.

• Prefix cache hit rate

  The share of prompt tokens found already in the cache rather than prefilled.

• Prefill

  The forward pass over the whole prompt at once, producing the KV cache for every token and the first output token.

• sources/
• Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse (Heo et al., 2026)
• topics/
• #kv-cache#prefix-cache#routing