glossary/compute/kernel-launch-overhead
Kernel launch overhead
The fixed cost of asking the GPU to run a kernel: the driver call, the scheduling, the queue. It is a few microseconds regardless of how much work the kernel does, so it matters exactly when kernels are small, which is decode at small batch. A step of many tiny kernels can spend more time launching than computing. CUDA graphs and fusion are the two ways to pay it less often.
a few microseconds
Per launch; against a kernel that runs for ten, that is a large share of the step.
Related
- glossary/
- CUDA graphs
Recording a sequence of kernel launches once and replaying the whole sequence with a single call.
- Kernel fusion
Combining several operations into one kernel so intermediate results stay in registers or on-chip memory instead of making a round trip to HBM.
- Decode
Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token.
- Nsight
NVIDIA's profilers.
- sources/
- topics/