Explainer
The hit rate is the workload
A coding agent sends its entire history with every turn: system prompt, tool definitions, every file it read, every reply. The median turn in SemiAnalysis’s AgentX traces is 88K tokens in and 413 out. Almost nothing is new. Whether the old part is still in memory is the whole cost of the turn.
The KV cache page showed that history costs 320 KiB a token to hold. The prefill page showed that recomputing it is compute-bound and slow. Put those together and an agent turn has three possible prices, depending on where its prefix is when it arrives. Add sessions below and watch which price the turns start paying.
20% of turns found nothing and paid 10.11 s to rebuild 88K tokens of history from scratch. Add host memory and watch the same misses become copies instead.
Three prices, three orders of magnitude apart. If the prefix is in HBM the turn simply starts; the cache read is part of the step it was always going to take. If it was spilled to host memory, 29.5 GB comes back over the host link, about two and a half seconds at the bandwidth from the memory hierarchy. If it was evicted, 88K tokens are prefilled from scratch, ten seconds on four cards at the roofline’s peak. Nothing about the model changed between those three cases. Only where the bytes were.
Concurrency does not degrade the cache. It breaks it. One hundred and eighty gigabytes holds five 88K sessions through a whole conversation, and they hit every time. Add a sixth and the growing histories cross the budget within a couple of turns; the least-recently-used session is evicted to make room, and since every session takes turns, the one evicted is the one about to be needed. Past the budget, hit rate falls off a cliff rather than a slope, which is why the article found 91% on the card with more HBM and 73% on the one with less, at the same workload.
Offload is a tier, not a fix, and the real levers are elsewhere. Host memory turns ten-second recomputes into two-second copies, until it fills too. The engine changes the article catalogues are about not spilling in the first place: keep the shared head of the context and drop the tail rather than evicting whole sessions, which took one implementation’s hit rate from 5.6% to 96%; and route a returning session to the card that already holds its cache instead of whichever is free. Both are scheduling decisions. Neither needs new hardware.
What this model leaves out
Caches here are whole sessions, evicted all or nothing. Real engines page the cache in blocks and can keep a shared prefix while dropping a session’s tail, which is exactly the selective retention described above. That makes real hit rates better than these at the same concurrency, and it is the point.
Sessions share nothing. In practice every agent on a deployment starts with the same system prompt and tool definitions, and a radix-structured cache stores that head once for all of them. The sharing works in caching’s favour and is not modelled.
The host link is the 12 GB/s figure from the A100 hierarchy, which is a floor. Current interconnects move cache back several times faster, so the copy price is pessimistic; the recompute price is not, since prefill is compute-bound and the compute is what it is. The cards are 80 GB for capacity but priced at A100 40GB bandwidth and peak, for the same reason as every other page: those figures are published.
Each miss is also charged only to its own turn. On a shared engine a ten-second prefill stalls every other session’s decode while it runs, which the prefill page covers and which makes the cost of a miss larger than shown.