feed/tags/networking
networking
8 glossary terms, and one reading ordered as a path, each with one line on why it earns your time.
- KV cache offloading
Spilling evicted KV cache to host DRAM, or further to SSD or a remote store, instead of throwing it away.
- Tensor parallelism
Splitting each weight matrix across GPUs so that every layer's matmul runs on all of them at once, with an all-reduce to combine the partial results.
- Pipeline parallelism
Putting different layers on different GPUs and passing activations along the chain.
- Expert parallelism
Placing the experts of a mixture-of-experts model on different GPUs, so each card holds a few experts rather than a slice of all of them.
- Wide expert parallelism
Expert parallelism stretched across many GPUs, often a whole rack, so each card holds one or two experts and the batch of tokens reaching each is large enough to make its GEMM efficient.
- All-to-all
The collective in which every GPU sends a different piece of data to every other GPU.
- NVLink
NVIDIA's GPU-to-GPU interconnect, an order of magnitude faster than PCIe, and the switch fabric that joins every GPU in a node or rack to every other at full rate.
- Interconnect bandwidth
Bytes per second between GPUs, whether over NVLink inside a node, InfiniBand or Ethernet between nodes, or PCIe to the host.
Current work, at the depth practitioners actually argue about.