glossary/tooling/tensorrt-llm
TensorRT-LLM
NVIDIA's inference library for its own GPUs: compiled engines with hand-tuned kernels, the first place FP8 and FP4 land, in-flight batching, paged KV cache, and the wide expert parallelism and disaggregated serving that its newest racks are built for. It is the peak of what the hardware can do and the most work to operate, since engines are built per model, per precision and per GPU.
FP4 first
The precision each new generation ships with, where the vendor stack usually leads the open engines by months.
Related
- glossary/
- FP8
An 8-bit float in two flavours: E4M3 with more precision for weights and activations, E5M2 with more range for gradients.
- FP4
A 4-bit float with sixteen representable values, usable only with fine-grained scaling: a shared scale factor per small block of values restores the range the format lacks.
- Wide expert parallelism
Expert parallelism stretched across many GPUs, often a whole rack, so each card holds one or two experts and the batch of tokens reaching each is large enough to make its GEMM efficient.
- Disaggregated serving
Running prefill and decode on separate pools of GPUs and shipping the KV cache between them.
- sources/
- topics/