How model serving actually behaves once it meets production traffic.
A community resource for inference engineering. Measurement you can reproduce, benchmarks built around real workloads rather than leaderboard rank, and the reading that explains your latency and your bill.
Sections
- feed/
Papers, posts and release notes worth the read, with a note on why.
- explainers/
Simulations you can run for the concepts that serving actually turns on.
- glossary/
Sixty terms, each with one number that pins it to a real machine.
Latest in the feed
all entries- paperCross-model KV cache transfer in LLM families: a closed-form linear mapping for prefill reuse
Every model swap in a cascade or a router throws the prefill away, because the bigger sibling cannot read the smaller one's cache. This paper fits a per-head ridge regression from the small model's keys and values to the large one's, with RoPE stripped so the map is position-free, on 500 calibration sequences. Four of six pairs keep 73 to 98% of standalone accuracy and the map runs 2.7 to 25x faster than re-prefill; two pairs fall apart, which is the honest part. Prefix caching across models, not just across requests.
Taekyung Heo, Rasoul Shafipour, Ritchie Zhao and others#kv-cache#prefix-cache#prefill#routing#cost - paperTaming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
Names the generation stall and removes it: prefill in chunks that ride inside decode steps under a per-step token budget, so a long prompt never freezes the batch. 2.6x the serving capacity of vLLM for Mistral 7B on one A100 under tail-latency SLOs, 3.7x for Yi-34B on two. OSDI 2024; every engine has it now.
Amey Agrawal, Nitin Kedia, Ashish Panwar and others#chunked-prefill#prefill#scheduling#latency