glossary/precision/activation-quantization
Activation quantization
also W8A8, SmoothQuant
Quantizing the activations flowing between layers as well as the weights, so the matmul itself runs in low precision on the tensor cores. This is what turns a memory saving into a compute saving: W8A8 runs at the INT8 or FP8 rate, twice BF16. Activations have outliers that resist naive rounding; SmoothQuant migrates that difficulty into the weights, which quantize more easily.
2x compute
The tensor core rate gain from 8-bit matmuls, on top of half the bytes, which is why it helps prefill where weight-only does not.
Related
- glossary/
- Weight-only quantization
Storing the weights in four or eight bits and converting them back to 16 on the way into the multiply, leaving activations at full precision.
- FP8
An 8-bit float in two flavours: E4M3 with more precision for weights and activations, E5M2 with more range for gradients.
- Compute-bound
Limited by arithmetic throughput; the memory system keeps up and the tensor cores are the bottleneck.
- Tensor cores
The GPU units that do small dense matrix multiplies in one instruction, at far higher rate than the general-purpose cores.
- sources/
- topics/