glossary/memory/working-set

Working set

The amount of KV cache the active sessions need resident to keep hitting. For a chat deployment it is small and turns over fast; for agents with long histories it is the sum of every live context, and it grows every turn as replies are appended. Once the working set exceeds the cache, the hit rate falls off a cliff rather than a slope, because sessions evict each other in rotation.

See it happen

• explainers/
• intermediateThe hit rate is the workload

  Agent sessions taking turns against one card's memory. Add sessions and watch each turn's cost jump from nothing to ten seconds as its history is evicted.

Related

• glossary/
• Prefix cache hit rate

  The share of prompt tokens found already in the cache rather than prefilled.

• Cache eviction

  Dropping stored KV cache to make room for new requests.

• KV cache

  The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.

• sources/
• The Working Set Model for Program Behavior (Denning, 1968)
• topics/
• #memory#agents#long-context