Skip to content
inference.academy

Explainer

Every layer stops to agree, twice

Split a 70B model across eight GPUs and each one reads an eighth of the weights per token, which is the entire point. But an eighth of a matrix multiply is a partial answer, and the eight partial answers have to be summed before the next layer can start. That sum is an all-reduce, there are two per layer, and at batch one each moves 16 KB. The bytes are free. The 160 stops are not.

Decode is memory-bound, so dividing the weights across N GPUs divides the time each spends reading them by N. Tensor parallelism is how a model that does not fit on one card runs, and how a model that does fit runs faster. The price is paid in collectives. Below, one decode step at every degree of tensor parallelism, with the communication drawn in.

Model: Llama 3.3 70B
Batch: 8 requests
Interconnect: NVLink
TP 145.0 ms if it fit; 80 GB is not enoughTP 224.5 ms if it fit; 160 GB is not enoughTP 413.3 ms, 601 tok/s, 16% commTP 87.7 ms, 1.0K tok/s, 27% comm
each GPU reads its shard of weights and cacheall-reduce latency floor, 160 per stepall-reduce bytes over the link
TP 8 against TP 4
1.7x
faster for 2x the GPUs
Communication at TP 8
27%
of a 7.7 ms step, 2.0 ms of it pure latency
One all-reduce message
128 KB
batch x 8192 hidden x 2 bytes
All-reduces per token
160
two per layer, 12.5 us floor each

At batch 8 each all-reduce carries 128 KB, which crosses NVLink in well under a microsecond. Almost all of the 2.1 ms of communication at TP 8 is the 160 launches themselves. Going from TP 4 to TP 8 buys 1.7x for 2x the GPUs, and the missing part is a fixed tax that no faster link can remove.

Per-GPU throughput against TP, both links, this batch
perfect scaling keeps this flat0150TP 1TP 2TP 4TP 8tokens per second per GPUNVLinkPCIe
H100 SXM GPUs, BF16, 4096 tokens of context per request. Local time is the roofline: each GPU’s shard of weights and cache against 3.35 TB/s, or its share of the arithmetic against 989 TFLOPS, whichever is slower. An all-reduce is a latency floor of 12.5 us plus a ring’s 2(N-1)/N of the message over the link, fitted to TokenWeave’s measurements of vLLM’s all-reduce on 8 H100 SXM: 1 MB in 16.3 us, 16 MB in 74.9 us, 128 MB in 500.5 us. NVLink is 450 GB/s per direction; PCIe Gen5 x16 is 64 GB/s and is given the same floor, which flatters it. A configuration is marked as not fitting when the shard exceeds 90% of 80 GB.

At small batch the all-reduce is a launch, not a transfer. One all-reduce carries the batch’s activations: batch times hidden size times two bytes, so 16 KB for one request through a 70B model and 2 MB at batch 128. NVLink moves 2 MB in a few microseconds. What it cannot do is start the collective for free: TokenWeave measured vLLM’s all-reduce on eight H100s at 16 microseconds for a 1 MB message and 500 for 128 MB, which is a floor of about 12.5 microseconds plus the bytes at NVLink speed. Eighty layers, two each, is 160 floors, two milliseconds a token whatever the batch. That is why per-GPU throughput falls as TP widens, and why the 8B model at TP 8 is spending more than half its step waiting to agree.

The link starts to matter when the batch is big enough to be worth serving. At batch 256 the message is 4 MB, the ring moves seven-eighths of it twice per GPU, and PCIe’s 64 GB/s per direction turns that into most of the step. This is the regime the Flash Communication paper saw on L40s over PCIe, communication up to 65% of latency, against about 20% on A100s with NVLink. On NVLink at that batch the step is also compute-bound enough that the shards have real arithmetic to hide the transfer behind, which is what the overlap work is about.

What the current work removes is microseconds, and it counts. TensorRT-LLM’s multishot uses NVSwitch multicast to make the all-reduce two steps regardless of GPU count, up to three times lower latency than a ring. NCCL 2.27’s symmetric memory claims up to nine times lower latency for small messages. A 2026 paper from ETH and NVIDIA gets a one-shot all-reduce on GB200 to 2.4 microseconds against a physical bound of 1.4, and reports that each microsecond taken off the collective is about 0.9% off the cost of serving Llama 70B at TP 4. The fused kernels that overlap the collective with the norm that follows it, TokenWeave and Cohere’s megakernel among them, are the other route to the same milliseconds.

What this model leaves out

Prefill. The same two all-reduces per layer happen there with the whole prompt’s activations in the message, tens of megabytes, so prefill is where bandwidth matters and where the interconnect decides time to first token for long prompts. This page prices decode only.

Pipeline and expert parallelism. Pipeline stages pass activations point to point, once per stage boundary rather than twice per layer, at the cost of bubbles. Expert parallelism replaces the all-reduce with two all-to-alls per MoE layer, dispatch and combine, whose cost DeepEP measures in the tens to hundreds of microseconds and which is a page of its own.

The PCIe floor is a guess in the interconnect’s favour. The 12.5 microsecond latency was measured on NVSwitch; a ring over PCIe without a switch is worse, and the page gives it the same floor so that only the bandwidth differs. Real PCIe systems also have to cross a CPU socket or a PLX switch, and neither is here. Multi-node tensor parallelism over InfiniBand is not here either; the measured floor there is about 33 microseconds.

Read next