glossary/precision/fp8
FP8
also E4M3, E5M2
An 8-bit float in two flavours: E4M3 with more precision for weights and activations, E5M2 with more range for gradients. Hopper and later run tensor core matmuls in it at twice the BF16 rate, and it halves the bytes of weights and cache, so it helps both a compute-bound prefill and a memory-bound decode. With per-tensor or per-block scaling it holds quality on most models without retraining.
70 GB
For Llama 3 70B's weights in FP8, halving the read per decode step and the floor on inter-token latency with it.
See it happen
Related
- glossary/
- BF16
A 16-bit float with the same 8-bit exponent as FP32 and only 7 bits of mantissa.
- FP4
A 4-bit float with sixteen representable values, usable only with fine-grained scaling: a shared scale factor per small block of values restores the range the format lacks.
- Tensor cores
The GPU units that do small dense matrix multiplies in one instruction, at far higher rate than the general-purpose cores.
- KV cache quantization
Storing the cached keys and values in 8 or 4 bits instead of 16.
- sources/
- topics/