Explainer
Compute the system prompt once
Every request to a deployment begins the same way, and every turn of a conversation begins with the turn before it. A cache that is addressed by prefix can hold that beginning once and let every request point at it. Step through the requests below and watch how little each one actually has to compute.
Attention over a prefix depends only on the prefix. The keys and values for “You are a helpful support agent” are the same whoever types the question after it, so the block of cache holding them can be shared by every request that starts that way. The same is true along time: the reply the model wrote last turn was written into the cache as it decoded, and the next turn inherits it.
Nothing is cached yet. The first request from each system prompt pays for it in full; everyone after them does not.
Two kinds of sharing, one mechanism. Across users, the shared thing is the head: system prompt, tool definitions, few-shot examples, whatever the deployment prepends to everyone. Along a conversation, the shared thing is everything up to the newest message. Both are the same operation on the cache: walk the tree of stored prefixes as far as it matches, and compute only what is left. In the run above that is a message of a few dozen tokens against a context of thousands.
It cuts prefill, and it cuts memory. Prefill is the obvious saving: the first token arrives after a few dozen tokens of compute rather than a few thousand. The quieter saving is memory. Without sharing, eight users of the same 2K system prompt hold eight copies of it, 16K tokens of cache for 2K tokens of information. With sharing they hold one, and the cache page’s budget goes further than its arithmetic says. Paging is what makes this possible: a block can belong to many requests only once cache is addressed by block.
There is a floor, and it is the weights. With everything cached, a turn still costs about 23 milliseconds before its first token on these four cards. That is not the cache missing; it is the prefill page’s memory-bound floor, the time to read the weights once, which no amount of caching removes. Prefix caching turns time-to-first-token from a function of context length into a constant, and the constant is the model.
What this model leaves out
Nothing is ever evicted here. Every prefix stays as long as the run does, which is the ideal case. The prefix cache page is the same idea under memory pressure, where the hit rate is decided by how many sessions are competing.
Matching is by whole segment. Real caches match at block granularity, so a prefix that diverges mid-block loses that block; and they hash blocks by content, which is what lets two users with byte-identical prompts share without anyone declaring it. A prompt that differs by one early token, a timestamp or a user name at the top, shares nothing after it, which is why deployments put the variable part last.
Replies are appended for free, which is true, and treated as the same cost as prompt tokens to hold, which is also true. What is not shown is that a cached reply is only useful if the same user comes back before it is evicted, which is a scheduling question rather than a caching one.