glossary/scale/wide-ep
Wide expert parallelism
also Wide-EP, large-scale EP
Expert parallelism stretched across many GPUs, often a whole rack, so each card holds one or two experts and the batch of tokens reaching each is large enough to make its GEMM efficient. At small scale, an expert sees few tokens per step and runs at memory speed; spread wide, the same expert serves tokens from the entire rack. It moves the bottleneck to the all-to-all, which is why it arrived with NVLink-connected 72-GPU domains.
72 GPUs
In one NVLink domain on GB200 NVL72, the scale at which the term became a product feature.
Related
- glossary/
- Expert parallelism
Placing the experts of a mixture-of-experts model on different GPUs, so each card holds a few experts rather than a slice of all of them.
- All-to-all
The collective in which every GPU sends a different piece of data to every other GPU.
- MoE routing
In a mixture-of-experts layer, a small gating network scores each token against every expert and sends it to the top few.
- NVLink
NVIDIA's GPU-to-GPU interconnect, an order of magnitude faster than PCIe, and the switch fabric that joins every GPU in a node or rack to every other at full rate.
- sources/
- topics/