glossary/serving/kv-cache-aware-routing
KV-cache-aware routing
also prefix-aware routing, cache-aware load balancing, KV router
A load balancer that knows which replica's memory holds which prefixes, and routes a request to the one with the longest match instead of the next one in line. Prefix caches are per replica, so a router that spreads a session's turns evenly across N replicas finds the cached history about one time in N. Sticky sessions fix that and cannot rebalance. Cache-aware routers, SGLang's router, NVIDIA Dynamo's KV router, the Kubernetes inference gateway and llm-d, keep an index of prefixes per worker and fall back to the shortest queue when load drifts past a threshold.
11% to 93%
Prefix cache hit rate on eight replicas under round-robin against prefix-aware routing, measured by llm-d on Qwen2.5-7B; mean time to first token fell from 19 seconds to under one.
See it happen
Related
- glossary/
- Prefix caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.
- Prefix cache hit rate
The share of prompt tokens found already in the cache rather than prefilled.
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- RadixAttention
SGLang's prefix cache: every stored sequence is a path in a radix tree keyed by tokens, so a new request walks the tree as far as its prompt matches and computes only the remainder.
- sources/
- SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer (LMSYS, 2024)
- KV-Cache Wins You Can See (llm-d, 2025)
- How llm-d Prefix-Cache Routing Made Qwen 7B on EKS 2.3x Faster
- topics/