glossary/precision/fp4
FP4
also NVFP4, MXFP4, E2M1
A 4-bit float with sixteen representable values, usable only with fine-grained scaling: a shared scale factor per small block of values restores the range the format lacks. Blackwell runs it on tensor cores at twice the FP8 rate. It quarters the weight bytes of BF16, which for a memory-bound decode is a fourfold cut in the floor per token, at a quality cost that depends heavily on the model and the calibration.
35 GB
For Llama 3 70B's weights at half a byte each; the KV cache, if left in BF16, then dominates memory at long context.
See it happen
Related
- glossary/
- FP8
An 8-bit float in two flavours: E4M3 with more precision for weights and activations, E5M2 with more range for gradients.
- BF16
A 16-bit float with the same 8-bit exponent as FP32 and only 7 bits of mantissa.
- Weight-only quantization
Storing the weights in four or eight bits and converting them back to 16 on the way into the multiply, leaving activations at full precision.
- Tensor cores
The GPU units that do small dense matrix multiplies in one instruction, at far higher rate than the general-purpose cores.
- sources/
- Introducing NVFP4 for Efficient and Accurate Low-Precision Inference (NVIDIA, 2025)
- Microscaling Data Formats for Deep Learning (Rouhani et al., 2023)
- topics/