feed/tags/quantization
quantization
7 glossary terms, and 2 readings ordered as a path, each with one line on why it earns your time.
- BF16
A 16-bit float with the same 8-bit exponent as FP32 and only 7 bits of mantissa.
- FP8
An 8-bit float in two flavours: E4M3 with more precision for weights and activations, E5M2 with more range for gradients.
- FP4
A 4-bit float with sixteen representable values, usable only with fine-grained scaling: a shared scale factor per small block of values restores the range the format lacks.
- Weight-only quantization
Storing the weights in four or eight bits and converting them back to 16 on the way into the multiply, leaving activations at full precision.
- Activation quantization
Quantizing the activations flowing between layers as well as the weights, so the matmul itself runs in low precision on the tensor cores.
- KV cache quantization
Storing the cached keys and values in 8 or 4 bits instead of 16.
- TensorRT-LLM
NVIDIA's inference library for its own GPUs: compiled engines with hand-tuned kernels, the first place FP8 and FP4 land, in-flight batching, paged KV cache, and the wide expert parallelism and disaggregated serving that its newest racks are built for.
Current work, at the depth practitioners actually argue about.
- articleOptimizing on-device inference for Apple silicon
The roofline on one laptop. Decode hits 90% of the chip's sustained weight-read rate and removing the arithmetic changes nothing, so speculative decoding comes out 18% slower at batch 1.
Perplexity Engineering#decode#prefill#quantization#kernels#hardware - paperTOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
You are billed for a model you cannot inspect. A 258-byte proof per 32 tokens that catches a swapped model, an altered prompt, or quietly reduced precision.
Jack Min Ong and colleagues#verifiability#trust#quantization