glossary/serving/continuous-batching
Continuous batching
also iteration-level scheduling, in-flight batching
Scheduling at the granularity of a single decode step instead of a whole batch. When one request finishes, its slot is filled by the next waiting request at the very next step, so the batch never drains to its slowest member. Introduced by Orca. It is the reason a GPU can stay busy under a mix of short and long answers, and the reason a KV cache must be allocated and freed per request rather than per batch.
one step
How long a freed slot stays empty, versus the length of the longest answer under static batching.
See it happen
Related
- glossary/
- Static batching
Collecting a batch of requests, running them together until every one has finished, then collecting the next.
- Request scheduling
Deciding, each step, which waiting requests join the batch and which running ones stay.
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- PagedAttention
Storing the KV cache in fixed-size blocks that need not be contiguous, with a per-request table mapping its logical positions to physical blocks, exactly as an operating system pages virtual memory.
- sources/
- topics/