glossary/serving/tpot
Time per output token
also TPOT
The average time each output token took, computed as total generation time divided by tokens generated, excluding the first. It is inter-token latency smoothed into one number per request. Benchmarks report it because it is stable; users feel the individual gaps, so a request with a fine TPOT can still stutter if a preemption or a long prefill from another request lands mid-stream.
p99, not mean
The number to hold a serving stack to, since the mean hides the stalls that scheduling causes.
See it happen
Related
- glossary/
- Inter-token latency
The gap between one output token and the next, once streaming has started.
- Decode
Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token.
- Service level objective
A latency promise stated as a percentile: the first token within 500 ms for 99% of requests, or each following token within 50 ms.
- Goodput
Throughput counted only for requests that met their latency targets.
- sources/
- topics/