Explainer
You are not renting the weights. You are renting the cache.
Model weights load once and never move. The KV cache is allocated per request, grows with every token, and is what runs out first. Fill the memory below until it breaks.
Attention needs the key and value vectors for every previous token. Recomputing them each step would be quadratic work, so they are kept. That store is the KV cache, it belongs to a single request, and it lives in the same memory as the weights.
Room for 16 concurrent requests at 32K context. You are using 16.
The weights are a fixed cost and the cache is a variable one. Llama 3 70B in BF16 is 140 GB of weights, which is why it needs more than one 80 GB card before it serves a single user. Everything left after that is the budget you have to sell. On four cards that is 180 GB, and at 32K context each concurrent request wants 10.7 GB of it. Sixteen users and you are full.
Context length is the multiplier nobody prices in. Cache per request is linear in context, so moving from 8K to 128K makes each user sixteen times more expensive to hold. A 128K context window advertised on a model card is not a feature you get for free, it is a promise to spend 40 GiB of memory per user who takes you up on it.
This is the ceiling the batching page ran into. Continuous batching keeps slots full, but it cannot create slots. Batch size is capped by how many caches fit alongside the weights, which is why every serious lever in serving, grouped-query attention, quantized caches, paging, offload, disaggregation, is ultimately an argument about this one number. Drop the cache to FP8 above and watch the same hardware hold roughly twice the users.
Where that memory actually sits, and what it costs to touch
“It fits” is only half the question. Memory on a GPU is a hierarchy, and the tier that holds your weights and caches is not the tier that does the arithmetic. Every forward pass streams chunks down from HBM into a sliver of on-chip SRAM, computes, and throws them away. Capacity decides whether you can serve someone. Bandwidth decides how fast.
At batch 1, one token pays to read all 140 GB of weights by itself. Raise the batch and that same read is split.
Note the ratio between those tiers. SRAM is roughly two thousand times smaller than HBM and about twelve times faster. Nothing of consequence stays there. The host link below is another two orders of magnitude slower again, which is why weights are loaded once at startup and why anything that forces a reload, a cold start, a model swap, is measured in seconds rather than microseconds.
This is why batching pays at all. Generating one token requires reading every weight out of HBM. At batch 1 a single token carries that entire cost. At batch 64 the same read is split sixty-four ways, and cost per token collapses by roughly fifty times while the bytes moved barely change. The arithmetic did not get cheaper; the traffic got shared.
And it is why the cache eventually bites twice. Cache does not merely occupy space, it is re-read every step for every active request. Push context and concurrency far enough and the green half of that bar overtakes the grey: you stop paying to move the model and start paying to move everyone’s history. That is the point where longer contexts stop being a memory problem and become a throughput one.
What this model leaves out
Real deployments never get the whole card. Activations, CUDA context, fragmentation and the runtime itself take a slice before any cache is allocated, so treat the concurrency figure as a ceiling you will not reach rather than a target.
Every request is also given a full context window here. In practice most requests are far shorter than the maximum, engines allocate in pages rather than reserving the worst case, and shared prefixes are stored once across requests instead of per user. That is exactly what PagedAttention and prefix caching are for, and it is why real concurrency beats this arithmetic.
Units are worth stating plainly. Cache per token is exactly 320 KiB for Llama 3 70B, which is the figure usually quoted as “320 KB”. This page divides in decimal GB, matching how cards are sold, so totals differ by a few percent from slides that mix the two conventions. The physics is identical either way.