feed/tags/latency
latency
3 explainers you can run, 8 glossary terms, and one reading ordered as a path, each with one line on why it earns your time.
- beginnerA batch is only as fast as its slowest member
Run the same fourteen requests under static and continuous batching, and watch where a GPU's idle time actually comes from.
- intermediateTwo phases, two bottlenecks
Move sequence length and batch size across a roofline and watch the limit flip from compute to memory bandwidth, the reason disaggregation exists.
- beginnerCompute the system prompt once
Eight users, one system prompt, several turns each. Step through the requests and watch what each one actually computes once the cache is addressed by prefix.
- Time to first token
The delay between a request arriving and the first token of its answer leaving.
- Inter-token latency
The gap between one output token and the next, once streaming has started.
- Time per output token
The average time each output token took, computed as total generation time divided by tokens generated, excluding the first.
- Goodput
Throughput counted only for requests that met their latency targets.
- Service level objective
A latency promise stated as a percentile: the first token within 500 ms for 99% of requests, or each following token within 50 ms.
- Chunked prefill
Splitting a long prompt's prefill into pieces of a few thousand tokens and running each piece in the same step as the batch's decode work.
- Speculative decoding
Having a small draft model guess several tokens ahead, then checking all of them with the large model in one forward pass.
- Kernel launch overhead
The fixed cost of asking the GPU to run a kernel: the driver call, the scheduling, the queue.
The mental models the rest of the field takes for granted.