glossary/precision/bf16
BF16
also bfloat16, brain floating point
A 16-bit float with the same 8-bit exponent as FP32 and only 7 bits of mantissa. It keeps FP32's range, so nothing overflows, and gives up precision, which transformers tolerate. It is the baseline precision for weights and the KV cache on this site: two bytes per parameter, two bytes per cached value. Every lower precision is measured as a fraction of it.
140 GB
For Llama 3 70B's weights at two bytes each, and 320 KiB of cache per token.
See it happen
Related
- glossary/
- FP8
An 8-bit float in two flavours: E4M3 with more precision for weights and activations, E5M2 with more range for gradients.
- FP4
A 4-bit float with sixteen representable values, usable only with fine-grained scaling: a shared scale factor per small block of values restores the range the format lacks.
- Weight-only quantization
Storing the weights in four or eight bits and converting them back to 16 on the way into the multiply, leaving activations at full precision.
- KV cache quantization
Storing the cached keys and values in 8 or 4 bits instead of 16.
- sources/
- topics/