glossary/scale/nvlink
NVLink
also NVSwitch
NVIDIA's GPU-to-GPU interconnect, an order of magnitude faster than PCIe, and the switch fabric that joins every GPU in a node or rack to every other at full rate. It sets the boundary of tensor parallelism: within an NVLink domain the per-layer all-reduces are cheap; across it they are not. Domain size has grown from 8 GPUs per node to 72 per rack, and with it the models one domain can serve at once.
600 GB/s
Per A100 in total NVLink bandwidth; 900 GB/s on the H100, against 64 GB/s for a PCIe Gen5 x16 slot.
Related
- glossary/
- Tensor parallelism
Splitting each weight matrix across GPUs so that every layer's matmul runs on all of them at once, with an all-reduce to combine the partial results.
- Interconnect bandwidth
Bytes per second between GPUs, whether over NVLink inside a node, InfiniBand or Ethernet between nodes, or PCIe to the host.
- All-to-all
The collective in which every GPU sends a different piece of data to every other GPU.
- Wide expert parallelism
Expert parallelism stretched across many GPUs, often a whole rack, so each card holds one or two experts and the batch of tokens reaching each is large enough to make its GEMM efficient.
- sources/
- topics/