feed/tags/routing

routing

One explainer you can run, one glossary term, and one reading ordered as a path, each with one line on why it earns your time.

explainers/ 1 page

advancedCross-Model KV Cache Transfer in LLM Families
Escalate a request from a 14B model to its 32B sibling and price the handoff both ways: re-prefill the context, or map the small model's cache across with a per-head linear fit.

glossary/ 1 term

Cross-model KV cache transfer
Reusing the KV cache a small model built for a prompt as the starting cache of a larger model in the same family, so a cascade or router can escalate a request without the large model prefilling the context again.

advanced/ 1 entry

Current work, at the depth practitioners actually argue about.

paperCross-model KV cache transfer in LLM families: a closed-form linear mapping for prefill reuse
Every model swap in a cascade or a router throws the prefill away, because the bigger sibling cannot read the smaller one's cache. This paper fits a per-head ridge regression from the small model's keys and values to the large one's, with RoPE stripped so the map is position-free, on 500 calibration sequences. Four of six pairs keep 73 to 98% of standalone accuracy and the map runs 2.7 to 25x faster than re-prefill; two pairs fall apart, which is the honest part. Prefix caching across models, not just across requests.
Taekyung Heo, Rasoul Shafipour, Ritchie Zhao and others4 Aug 2026#kv-cache#prefix-cache#prefill#routing#cost

Related topics