Feed
Inference engineering is discussed in a dozen places and archived in none. This is the archive, ordered as a path rather than a pile, so you can start where you are. Subscribe over RSS.
Start here. No serving experience assumed.
- threadKV, Prefix, Prompt and Semantic Caching in LLMs, clearly explained
Four things routinely called 'caching' that solve four different problems. Untangling them is most of the way to reading anyone else's cost numbers.
Avi Chawla#kv-cache#prefix-cache#fundamentals - articleLLM Serving Fairness: No more noisy neighbors
Batching is what makes serving efficient and is exactly what lets one tenant ruin another's latency. A clear account of scheduling against that tension.
Manoj Govindassamy and Musa Talluzi#scheduling#multi-tenancy#batching
The mental models the rest of the field takes for granted.
- interactiveInteractive H100 architecture glossary
Rotate the die and hover the parts. Memory bandwidth stops being a number you quote and starts being a place on the chip.
Kyle Jeong, with definitions from Modal's GPU Glossary#gpu#hardware#hbm - talkProducing the World's Cheapest Tokens: A How-to Guide
Latency, cost, quality: you get two. The clearest statement of the trade every other decision on this site is downstream of.
Meryem Arik#cost#latency#inference-triangle#batching - paperEfficient Memory Management for Large Language Model Serving with PagedAttention
The KV cache is the scarce resource, not the weights. Everything about batching downstream of this follows from that one reframing.
#kv-cache#vllm#batching - paperOrca: A Distributed Serving System for Transformer-Based Generative Models
Where continuous batching comes from. Read it before you tune a max-batch-size you do not understand.
#batching#scheduling
Current work, at the depth practitioners actually argue about.
- toolNIXL: NVIDIA Inference Xfer Library
Disaggregation only pays if the KV cache moves between pools fast enough. This is the transfer layer underneath Dynamo doing that job.
#disaggregation#kv-cache#networking - paperSliding-window beats linear attention
A useful corrective: the older, simpler mechanism holds up against linear attention on long context, without the post-training bill.
Alexia Jolicoeur-Martineau and colleagues#attention#long-context#memory - articleSemi-Persistence: Blazing-Fast Model Swapping for Complex Scheduling
Cold start is the tax on serving many models from few GPUs. Splitting weights from the skeleton takes frontier-model wake-up from minutes to seconds.
Mert Hidayetoglu and colleagues#cold-start#vllm#gpu-sharing - articleThe case for disaggregated LLM serving
Disaggregation is usually argued for on principle. This argues it on conditions (scale, network, and traffic balance) and is honest that below them it does not pay.
Fergus Finn#disaggregation#prefill#decode#scale - articleGigatoken: making tokenisation care about the hardware
Nobody profiles the tokenizer. A hardware-aware rewrite finds 37x, which says something about where else nobody is looking.
Peter Bhabra#tokenization#cpu#simd - articleThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale
An agent that pauses for a tool call gets its cache evicted and pays to rebuild it. Scheduling the whole program instead of each request is the fix.
Hao Kang and colleagues#agents#kv-cache#scheduling - articleDefeating Nondeterminism in LLM Inference
Temperature zero is not determinism. A thousand identical requests gave eighty different completions, because kernel output depends on batch size, and batch size depends on who else is on the server.
Horace He#determinism#kernels#batching - paperTOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
You are billed for a model you cannot inspect. A 258-byte proof per 32 tokens that catches a swapped model, an altered prompt, or quietly reduced precision.
Jack Min Ong and colleagues#verifiability#trust#quantization - paperSGLang: Efficient Execution of Structured Language Model Programs
Prefix caching as a first-class scheduling primitive. If your agent workload is 80% cache hits, this is the paper explaining your bill.
#prefix-cache#sglang - paperSplitwise: Efficient Generative LLM Inference Using Phase Splitting
Prefill and decode want different hardware. The clearest statement of why disaggregation keeps winning.
#disaggregation#prefill#decode
Topics
- #batching 5
- #kv-cache 4
- #disaggregation 3
- #scheduling 3
- #decode 2
- #prefill 2
- #prefix-cache 2
- #vllm 2
- #agents 1
- #attention 1
- #cold-start 1
- #cost 1
- #cpu 1
- #determinism 1
- #fundamentals 1
- #gpu 1
- #gpu-sharing 1
- #hardware 1
- #hbm 1
- #inference-triangle 1
- #kernels 1
- #latency 1
- #long-context 1
- #memory 1
- #multi-tenancy 1
- #networking 1
- #quantization 1
- #scale 1
- #sglang 1
- #simd 1
- #tokenization 1
- #trust 1
- #verifiability 1