feed/tags/sglang

SGLang

One explainer you can run, 2 glossary terms, and one reading ordered as a path, each with one line on why it earns your time.

explainers/ 1 page

beginnerCompute the system prompt once
Eight users, one system prompt, several turns each. Step through the requests and watch what each one actually computes once the cache is addressed by prefix.

glossary/ 2 terms

RadixAttention
SGLang's prefix cache: every stored sequence is a path in a radix tree keyed by tokens, so a new request walks the tree as far as its prompt matches and computes only the remainder.
SGLang
A serving engine and frontend language built around RadixAttention, its radix-tree prefix cache, and a compressed finite-state machine for fast constrained decoding.

advanced/ 1 entry

Current work, at the depth practitioners actually argue about.

paperSGLang: Efficient Execution of Structured Language Model Programs
Prefix caching as a first-class scheduling primitive. If your agent workload is 80% cache hits, this is the paper explaining your bill.
Dec 2023#prefix-cache#sglang

Related topics