glossary/serving/static-batching

Static batching

Collecting a batch of requests, running them together until every one has finished, then collecting the next. Requests that finish early keep their slot and produce nothing while the longest one completes, so utilisation falls with the spread of answer lengths. It is the natural design for a training-style forward pass, and the thing continuous batching replaced for serving.

See it happen

• explainers/
• beginnerA batch is only as fast as its slowest member

  Run the same fourteen requests under static and continuous batching, and watch where a GPU's idle time actually comes from.

Related

• glossary/
• Continuous batching

  Scheduling at the granularity of a single decode step instead of a whole batch.

• Throughput

  Tokens produced per second across all requests in flight, or requests completed per second.

• Request scheduling

  Deciding, each step, which waiting requests join the batch and which running ones stay.

• sources/
• How continuous batching enables 23x throughput in LLM inference (Anyscale, 2023)
• topics/
• #batching#scheduling