Explainer
A batch is only as fast as its slowest member.
That one sentence is the whole reason continuous batching exists. The schedule below is the same fourteen requests under both policies. Press play and watch where the gaps come from.
Each row is a slot the batch can hold. Each column is one iteration, a single forward pass. A request takes a few iterations to process its prompt, then produces one token per iteration until it stops. Requests do not stop at the same time, and that is where the trouble starts.
What the person waiting sees
Why is my p99 latency spiking?
Summarise this incident report
Rewrite this paragraph to be shorter
| Scheduler | Slots doing work | Iterations to drain | Mean latency |
|---|---|---|---|
| static ← | 60.4% | 48 | 24.9 iters |
| continuous | 87.9% | 33 | 18.3 iters |
Static batching admits a group and waits for all of it. A slot whose request finished early cannot take new work, because the batch is a fixed unit, so it sits idle until the longest request in the group is done. Those grey cells are a GPU you are paying for and not using. The more variance in output length, the worse it gets, and real traffic has enormous variance.
Continuous batching schedules at the token level instead. When a request emits its last token, its slot is free on the very next iteration and the next queued request moves in. The batch stops being a group and becomes a population that is continuously topped up. Nothing about the model changed; the same work is done in fewer iterations purely because fewer slots sat empty.
Notice that the total coloured area is identical in both modes: the same requests do the same work. Only the whitespace changes. That is the entire optimisation, and it is why the technique also improves latency rather than trading against it: a queued request starts as soon as any slot frees, instead of waiting for a whole batch to drain.
Watch the last request in the streaming panel, the one that arrives with nobody free to take it. Under static batching it sits at “waiting for a slot” through entire batches before a single token reaches whoever asked. Flip the scheduler and its first token arrives roughly twice as early, off the same hardware doing the same work. Utilisation is the operator’s number; time to first token is the one the person waiting actually feels, and here they move together instead of against each other.
What this model leaves out
Prefill here occupies whole iterations, one cell at a time. Real engines chunk prefill and mix it with decode in the same iteration, which is why you will see “chunked prefill” as a tunable. The simulation also gives every slot equal cost, when in practice a long prompt costs far more to prefill than a token costs to decode.
Most importantly, slots are not free. Every concurrent sequence holds its own KV cache, and that memory is what actually caps your batch size, which is the subject of the next explainer.