glossary/compute/megakernel
Megakernel
also persistent kernel
A single GPU kernel that runs an entire decode step, with every operation expressed as tasks that the streaming multiprocessors pull from a queue and fine-grained counters standing in for kernel boundaries. A normal step launches dozens of kernels, and each boundary drains the pipeline, leaves idle SMs at the end of a wave and forbids the next operation from prefetching its weights. A megakernel removes the boundaries, which is worth most exactly where decode is worst: small batches, where the step is bound by reading the weights and every bubble is bandwidth thrown away.
39% to 62%
Of the bandwidth speed of light on one H100 for a 30B-parameter, 3.3B-active model at batch 1, vLLM against Cohere's megakernel; 185 to 292 tokens a second.
Related
- glossary/
- Kernel launch overhead
The fixed cost of asking the GPU to run a kernel: the driver call, the scheduling, the queue.
- Kernel fusion
Combining several operations into one kernel so intermediate results stay in registers or on-chip memory instead of making a round trip to HBM.
- CUDA graphs
Recording a sequence of kernel launches once and replaying the whole sequence with a single call.
- Memory-bound
Limited by how fast bytes can be read, not by how fast they can be operated on.
- Decode
Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token.
- sources/
- Inside the megakernel serving engine for North Mini Code (Cohere, 2026)
- Look Ma, No Bubbles! Designing a low-latency megakernel for Llama-1B (Hazy Research, 2025)
- cohere-ai/cohere-megakernel on GitHub
- topics/