glossary/scale/moe-routing

MoE routing

also mixture of experts, gating, top-k routing

In a mixture-of-experts layer, a small gating network scores each token against every expert and sends it to the top few. Only those experts run for that token, so compute per token is a fraction of the parameter count. For serving, the question is balance: if popular experts get most of the tokens, the cards holding them are the step's critical path while the rest wait. Load-balancing losses and capacity limits exist to spread the load.

Related

• glossary/
• Expert parallelism

  Placing the experts of a mixture-of-experts model on different GPUs, so each card holds a few experts rather than a slice of all of them.

• All-to-all

  The collective in which every GPU sends a different piece of data to every other GPU.

• Wide expert parallelism

  Expert parallelism stretched across many GPUs, often a whole rack, so each card holds one or two experts and the batch of tokens reaching each is large enough to make its GEMM efficient.

• GEMM

  General matrix multiply, the operation the projections and feed-forward layers of a transformer reduce to.

• sources/
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al., 2017)
• DeepSeek-V3 Technical Report (DeepSeek-AI, 2024)
• topics/
• #scale