glossary/serving/itl
Inter-token latency
also ITL, time between tokens
The gap between one output token and the next, once streaming has started. Each gap is one decode step, and a decode step at small batch is bound by how fast the weights can be read from memory, not by arithmetic. It rises with batch size, because every step now serves more requests, and with context length, because the cache read grows. Below about 50 ms it reads as fluent.
23.3 ms
Per token at batch 1 on four A100s for Llama 3 70B in BF16, about 43 tokens per second, set entirely by memory bandwidth.
See it happen
Related
- glossary/
- Time per output token
The average time each output token took, computed as total generation time divided by tokens generated, excluding the first.
- Decode
Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token.
- Time to first token
The delay between a request arriving and the first token of its answer leaving.
- Memory bandwidth
Bytes per second the GPU can move between its HBM and its compute units.
- sources/
- topics/