glossary/scale/all-to-all
All-to-all
also dispatch and combine
The collective in which every GPU sends a different piece of data to every other GPU. Expert parallelism needs two per MoE layer: one to dispatch each token's activations to the cards holding its experts, one to bring the results back. Unlike an all-reduce it is not a sum, so it cannot be tree-reduced, and its cost is set by the slowest link in the group. It is the communication that decides how wide expert parallelism can go.
two per MoE layer
Dispatch and combine, each crossing the interconnect for every token in the step.
Related
- glossary/
- Expert parallelism
Placing the experts of a mixture-of-experts model on different GPUs, so each card holds a few experts rather than a slice of all of them.
- Wide expert parallelism
Expert parallelism stretched across many GPUs, often a whole rack, so each card holds one or two experts and the batch of tokens reaching each is large enough to make its GEMM efficient.
- NVLink
NVIDIA's GPU-to-GPU interconnect, an order of magnitude faster than PCIe, and the switch fabric that joins every GPU in a node or rack to every other at full rate.
- Interconnect bandwidth
Bytes per second between GPUs, whether over NVLink inside a node, InfiniBand or Ethernet between nodes, or PCIe to the host.
- sources/
- topics/