glossary/scale/expert-parallelism
Expert parallelism
also EP
Placing the experts of a mixture-of-experts model on different GPUs, so each card holds a few experts rather than a slice of all of them. Tokens are routed to whichever card holds their chosen experts and the results routed back, an all-to-all exchange per MoE layer. It lets the total parameter count exceed any one node while the active count per token stays small, and it makes load balance across experts a hardware problem.
256 experts, 8 active
Per token in DeepSeek-V3, plus one shared expert; 671B parameters total, 37B used per token.
Related
- glossary/
- MoE routing
In a mixture-of-experts layer, a small gating network scores each token against every expert and sends it to the top few.
- All-to-all
The collective in which every GPU sends a different piece of data to every other GPU.
- Wide expert parallelism
Expert parallelism stretched across many GPUs, often a whole rack, so each card holds one or two experts and the batch of tokens reaching each is large enough to make its GEMM efficient.
- Tensor parallelism
Splitting each weight matrix across GPUs so that every layer's matmul runs on all of them at once, with an all-reduce to combine the partial results.
- sources/
- topics/