hardware

Move sequence length and batch size across a roofline and watch the limit flip from compute to memory bandwidth, the reason disaggregation exists.

The stacked DRAM on the GPU package that holds the weights and the KV cache.

Bytes per second the GPU can move between its HBM and its compute units.

A chart with arithmetic intensity on the x axis and attainable performance on the y, bounded by a sloped line from memory bandwidth and a flat line from peak compute.

Floating-point operations performed per byte moved from memory.

The arithmetic intensity at which a machine's memory ceiling and compute ceiling meet: peak FLOPS divided by bandwidth.

Limited by how fast bytes can be read, not by how fast they can be operated on.

Limited by arithmetic throughput; the memory system keeps up and the tensor cores are the bottleneck.

The GPU units that do small dense matrix multiplies in one instruction, at far higher rate than the general-purpose cores.

An 8-bit float in two flavours: E4M3 with more precision for weights and activations, E5M2 with more range for gradients.

A 4-bit float with sixteen representable values, usable only with fine-grained scaling: a shared scale factor per small block of values restores the range the format lacks.

Splitting each weight matrix across GPUs so that every layer's matmul runs on all of them at once, with an all-reduce to combine the partial results.

Expert parallelism stretched across many GPUs, often a whole rack, so each card holds one or two experts and the batch of tokens reaching each is large enough to make its GEMM efficient.

NVIDIA's GPU-to-GPU interconnect, an order of magnitude faster than PCIe, and the switch fabric that joins every GPU in a node or rack to every other at full rate.

Bytes per second between GPUs, whether over NVLink inside a node, InfiniBand or Ethernet between nodes, or PCIe to the host.

NVIDIA's inference library for its own GPUs: compiled engines with hand-tuned kernels, the first place FP8 and FP4 land, in-flight batching, paged KV cache, and the wide expert parallelism and disaggregated serving that its newest racks are built for.