How model serving actually behaves once it meets production traffic.
A community resource for inference engineering. Measurement you can reproduce, benchmarks built around real workloads rather than leaderboard rank, and the reading that explains your latency and your bill.
Sections
- feed/
Papers, posts and release notes worth the read, with a note on why.
- explainers/
Simulations you can run for the concepts that serving actually turns on.
- glossary/
Sixty terms, each with one number that pins it to a real machine.
Latest in the feed
all entries- paperTaming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
Names the generation stall and removes it: prefill in chunks that ride inside decode steps under a per-step token budget, so a long prompt never freezes the batch. 2.6x the serving capacity of vLLM for Mistral 7B on one A100 under tail-latency SLOs, 3.7x for Yi-34B on two. OSDI 2024; every engine has it now.
Amey Agrawal, Nitin Kedia, Ashish Panwar and others#chunked-prefill#prefill#scheduling#latency - paperAccelerating Large Language Model Decoding with Speculative Sampling
The same idea arrived at independently at DeepMind, on Chinchilla 70B with a 4B draft across a pod: two to two and a half times faster in a distributed setting where the draft has to be worth its own communication. The clearest write-up of the modified rejection sampling.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving and others#decode#latency#speculative-decoding - paperFast Inference from Transformers via Speculative Decoding
The paper that made a small model guessing ahead respectable: the target verifies the guesses in one pass and a rejection-sampling step keeps its output distribution exactly. Two to three times faster decode on T5-XXL with no retraining, and the expected tokens-per-step formula every later variant is measured against.
Yaniv Leviathan, Matan Kalman and Yossi Matias#decode#latency#speculative-decoding