glossary/serving/goodput
Goodput
Throughput counted only for requests that met their latency targets. A deployment can post a high token rate while half its users wait too long for a first token; goodput is the rate of requests served within both the first-token and per-token limits, which is the number that matches what was promised. Introduced for serving by the DistServe paper, where it motivates separating prefill from decode.
TTFT and TPOT
Both must be met for a request to count; the DistServe paper reports up to 7.4x more goodput under the same SLOs by disaggregating.
See it happen
Related
- glossary/
- Throughput
Tokens produced per second across all requests in flight, or requests completed per second.
- Service level objective
A latency promise stated as a percentile: the first token within 500 ms for 99% of requests, or each following token within 50 ms.
- Disaggregated serving
Running prefill and decode on separate pools of GPUs and shipping the KV cache between them.
- Time per output token
The average time each output token took, computed as total generation time divided by tokens generated, excluding the first.
- sources/
- topics/