glossary/compute/compute-bound
Compute-bound
Limited by arithmetic throughput; the memory system keeps up and the tensor cores are the bottleneck. Prefill of a long prompt is compute-bound, which is why its time grows linearly with prompt length and why a faster card, or a lower precision with more FLOPS, shortens time to first token. Batched decode reaches this regime only at very large batches, and then only if the cache for those batches fits.
above 208
FLOP per byte on the A100; a 4K prompt's prefill sits at about 4,000.
See it happen
Related
- glossary/
- Memory-bound
Limited by how fast bytes can be read, not by how fast they can be operated on.
- Prefill
The forward pass over the whole prompt at once, producing the KV cache for every token and the first output token.
- Tensor cores
The GPU units that do small dense matrix multiplies in one instruction, at far higher rate than the general-purpose cores.
- Roofline model
A chart with arithmetic intensity on the x axis and attainable performance on the y, bounded by a sloped line from memory bandwidth and a flat line from peak compute.
- sources/
- topics/