feed/tags/chunked-prefill

chunked prefill

One explainer you can run, 2 glossary terms, and one reading ordered as a path, each with one line on why it earns your time.

explainers/ 1 page

intermediateA long prompt should not stall everyone else
Sixteen streams mid-reply when a 16K prompt lands. Run its prefill alone and they freeze for two seconds; chunk it into decode steps and the longest gap is 30 milliseconds. The right chunk size is the ridge point.

glossary/ 2 terms

Chunked prefill
Splitting a long prompt's prefill into pieces of a few thousand tokens and running each piece in the same step as the batch's decode work.
Generation stall
A pause in every decoding stream in a batch while the scheduler runs another request's prefill as its own iteration.

advanced/ 1 entry

Current work, at the depth practitioners actually argue about.

paperTaming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
Names the generation stall and removes it: prefill in chunks that ride inside decode steps under a per-step token budget, so a long prompt never freezes the batch. 2.6x the serving capacity of vLLM for Mistral 7B on one A100 under tail-latency SLOs, 3.7x for Yi-34B on two. OSDI 2024; every engine has it now.
Amey Agrawal, Nitin Kedia, Ashish Panwar and othersMar 2024#chunked-prefill#prefill#scheduling#latency

Related topics