glossary/scale/tensor-parallelism
Tensor parallelism
also TP
Splitting each weight matrix across GPUs so that every layer's matmul runs on all of them at once, with an all-reduce to combine the partial results. It cuts per-GPU memory and, for memory-bound decode, multiplies effective bandwidth, since each card reads its slice of the weights in parallel. The all-reduces happen twice per layer and run at interconnect speed, so it works within an NVLink island and poorly beyond one.
6 TB/s
Aggregate weight-read bandwidth across four A100s under tensor parallelism, the number behind every latency floor on this site.
See it happen
Related
- glossary/
- Pipeline parallelism
Putting different layers on different GPUs and passing activations along the chain.
- NVLink
NVIDIA's GPU-to-GPU interconnect, an order of magnitude faster than PCIe, and the switch fabric that joins every GPU in a node or rack to every other at full rate.
- Interconnect bandwidth
Bytes per second between GPUs, whether over NVLink inside a node, InfiniBand or Ethernet between nodes, or PCIe to the host.
- Memory bandwidth
Bytes per second the GPU can move between its HBM and its compute units.
- sources/
- topics/