How model serving actually behaves once it meets production traffic.

A decoder that carries its final hidden state across every token, prompt and response alike, so depth grows with sequence length instead of being fixed by layer count. Worth reading here for what it costs rather than what it saves: recurrence prevents the decoder from processing all positions in parallel, so the prompt is consumed token by token and prefill as we know it stops existing, with only the causal encoder keeping sequence-level parallelism and a sliding window bounding the decoder cache. The author is explicit that this is a specification, reporting no throughput, latency or memory, and not claiming to reduce arithmetic or latency against a standard transformer. Read it as a design to argue with, not a result.

Long-horizon agents make the workload input-heavy, so the bytes of cache per token are the bill. V4.1-Flash attacks that in the architecture: the lower 20 of 40 layers run as an encoder and the decoder's global KV is projected from the encoder state, so prefill does about half the layer work; CSA2 shares main KV and indexer keys across layers, and Reindex layers rescore only a candidate pool the first full layer chose, so per-query cost stops growing with context; main KV is MXFP4 and the sliding-window cache is replayed on resume rather than stored. The result is 890 bytes per token of global KV in HBM at 1M context, a quarter of V4-Flash, an eighth in persistent storage. The absolute V4-Flash figure is not in the paper, so the ratios are what you get.

Every model swap in a cascade or a router throws the prefill away, because the bigger sibling cannot read the smaller one's cache. This paper fits a per-head ridge regression from the small model's keys and values to the large one's, with RoPE stripped so the map is position-free, on 500 calibration sequences. Four of six pairs keep 73 to 98% of standalone accuracy and the map runs 2.7 to 25x faster than re-prefill; two pairs fall apart, which is the honest part. Prefix caching across models, not just across requests.

A decode step on one H100 moves 6.6 GB of weights and 0.5 GB of cache, so its speed of light is about 470 tokens a second; vLLM gets 185 of them and the gaps between kernels are where the rest go. One persistent kernel with counter barriers, weight prefetch across tasks and idle SMs backfilled from the next layer gets 292, 1.58x at batch 1 and 1.32x at batch 8, with the code published. The clearest account yet of what a memory-bound step loses to launch boundaries.

Which provider fits your workload? 12,795 calls across 14 serving routes show how cost, first-token latency, generation speed and reliability change with prompt size and caching. Repeated prompts can change the cost ranking, including the comparison between automatic routing and pinned hosts. Explore the interactive charts, download the measurements, and run your own experiments with Compound.