glossary/serving/throughput
Throughput
also tokens per second, requests per second
Tokens produced per second across all requests in flight, or requests completed per second. It rises with batch size because a decode step reads the weights once for everyone, so serving more requests per step costs almost nothing extra until arithmetic becomes the limit. It is the number that decides cost per token, and it trades directly against latency: a bigger batch means a slower step.
512 requests
The batch at which decode on four A100s for Llama 3 70B with 1K contexts crosses from memory-bound to compute-bound, needing 182 GB of cache.
See it happen
Related
- glossary/
- Goodput
Throughput counted only for requests that met their latency targets.
- Continuous batching
Scheduling at the granularity of a single decode step instead of a whole batch.
- Inter-token latency
The gap between one output token and the next, once streaming has started.
- Arithmetic intensity
Floating-point operations performed per byte moved from memory.
- sources/
- topics/