feed/tags/decode
decode
One explainer you can run, 8 glossary terms, and 3 readings ordered as a path, each with one line on why it earns your time.
- Inter-token latency
The gap between one output token and the next, once streaming has started.
- Time per output token
The average time each output token took, computed as total generation time divided by tokens generated, excluding the first.
- Speculative decoding
Having a small draft model guess several tokens ahead, then checking all of them with the large model in one forward pass.
- Disaggregated serving
Running prefill and decode on separate pools of GPUs and shipping the KV cache between them.
- Memory bandwidth
Bytes per second the GPU can move between its HBM and its compute units.
- Decode
Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token.
- Memory-bound
Limited by how fast bytes can be read, not by how fast they can be operated on.
- CUDA graphs
Recording a sequence of kernel launches once and replaying the whole sequence with a single call.
Current work, at the depth practitioners actually argue about.
- articleOptimizing on-device inference for Apple silicon
The roofline on one laptop. Decode hits 90% of the chip's sustained weight-read rate and removing the arithmetic changes nothing, so speculative decoding comes out 18% slower at batch 1.
Perplexity Engineering#decode#prefill#quantization#kernels#hardware - articleThe case for disaggregated LLM serving
Disaggregation is usually argued for on principle. This argues it on conditions (scale, network, and traffic balance) and is honest that below them it does not pay.
Fergus Finn#disaggregation#prefill#decode#scale - paperSplitwise: Efficient Generative LLM Inference Using Phase Splitting
Prefill and decode want different hardware. The clearest statement of why disaggregation keeps winning.
#disaggregation#prefill#decode