glossary/compute/memory-bound
Memory-bound
also bandwidth-bound
Limited by how fast bytes can be read, not by how fast they can be operated on. The compute units idle waiting for memory. Decode is the canonical case: each step reads gigabytes of weights to do a handful of operations per byte. The fixes are all about bytes: fewer of them through quantization, more useful work per read through batching or speculative decoding, or a card with more bandwidth.
below 208
FLOP per byte on the A100; decode at batch 1 sits at about 1.
See it happen
Related
- glossary/
- Compute-bound
Limited by arithmetic throughput; the memory system keeps up and the tensor cores are the bottleneck.
- Decode
Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token.
- Memory bandwidth
Bytes per second the GPU can move between its HBM and its compute units.
- Roofline model
A chart with arithmetic intensity on the x axis and attainable performance on the y, bounded by a sloped line from memory bandwidth and a flat line from peak compute.
- sources/
- topics/