glossary/compute/cuda-graphs
CUDA graphs
Recording a sequence of kernel launches once and replaying the whole sequence with a single call. A decode step for a large model is hundreds of small kernels, and at batch 1 each takes a few microseconds to run and a few to launch, so launch overhead can be a large share of the step. Serving engines capture a graph per batch size and replay it, which is why decode latency at small batch is much better than a naive loop.
hundreds of launches
Per decode step for an 80-layer model, each costing single-digit microseconds of CPU time before the GPU sees it.
Related
- glossary/
- Decode
Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token.
- Kernel launch overhead
The fixed cost of asking the GPU to run a kernel: the driver call, the scheduling, the queue.
- Inter-token latency
The gap between one output token and the next, once streaming has started.
- vLLM
The open-source serving engine from UC Berkeley that introduced PagedAttention and made continuous batching with paged KV cache the default architecture.
- sources/
- topics/