feed/tags/batching
batching
One explainer you can run, 4 glossary terms, and 5 readings ordered as a path, each with one line on why it earns your time.
- Throughput
Tokens produced per second across all requests in flight, or requests completed per second.
- Continuous batching
Scheduling at the granularity of a single decode step instead of a whole batch.
- Static batching
Collecting a batch of requests, running them together until every one has finished, then collecting the next.
- Request scheduling
Deciding, each step, which waiting requests join the batch and which running ones stay.
Start here. No serving experience assumed.
The mental models the rest of the field takes for granted.
- talkProducing the World's Cheapest Tokens: A How-to Guide
Latency, cost, quality: you get two. The clearest statement of the trade every other decision on this site is downstream of.
Meryem Arik#cost#latency#inference-triangle#batching - paperEfficient Memory Management for Large Language Model Serving with PagedAttention
The KV cache is the scarce resource, not the weights. Everything about batching downstream of this follows from that one reframing.
#kv-cache#vllm#batching - paperOrca: A Distributed Serving System for Transformer-Based Generative Models
Where continuous batching comes from. Read it before you tune a max-batch-size you do not understand.
#batching#scheduling
Current work, at the depth practitioners actually argue about.