feed/tags/vllm

vLLM

One explainer you can run, 3 glossary terms, and 2 readings ordered as a path, each with one line on why it earns your time.

explainers/ 1 page

beginnerReserve for the worst case, or page
Forty requests of unknown length against one memory, under two allocators at once. The PagedAttention paper's 20 to 40% utilisation figure, reproduced rather than quoted.

glossary/ 3 terms

PagedAttention
Storing the KV cache in fixed-size blocks that need not be contiguous, with a per-request table mapping its logical positions to physical blocks, exactly as an operating system pages virtual memory.
Block table
The per-request list that says which physical block of KV cache holds each run of its tokens.
vLLM
The open-source serving engine from UC Berkeley that introduced PagedAttention and made continuous batching with paged KV cache the default architecture.

intermediate/ 1 entry

The mental models the rest of the field takes for granted.

paperEfficient Memory Management for Large Language Model Serving with PagedAttention
The KV cache is the scarce resource, not the weights. Everything about batching downstream of this follows from that one reframing.
Sept 2023#kv-cache#vllm#batching

advanced/ 1 entry

Current work, at the depth practitioners actually argue about.

articleSemi-Persistence: Blazing-Fast Model Swapping for Complex Scheduling
Cold start is the tax on serving many models from few GPUs. Splitting weights from the skeleton takes frontier-model wake-up from minutes to seconds.
Mert Hidayetoglu and colleaguesAug 2026#cold-start#vllm#gpu-sharing

Related topics