feed/tags/benchmark

benchmark

2 glossary terms, and one reading ordered as a path, each with one line on why it earns your time.

glossary/ 2 terms

Time per output token
The average time each output token took, computed as total generation time divided by tokens generated, excluding the first.
Goodput
Throughput counted only for requests that met their latency targets.

advanced/ 1 entry

Current work, at the depth practitioners actually argue about.

articleAgentX and InferenceX v3: does the CUDA moat hold up in agentic inference?
An agent turn is 88K tokens in and 413 out, so the benchmark is really a KV cache hit-rate benchmark: 91% on B300 at 384 sessions, 73% on B200 with the rest spilling to DRAM. Nvidia, AMD, five open models, 70 upstream PRs. Partly paywalled.
Cam Quilici, Bryan Shan, Alec Ibarra and othersAug 2026#agents#prefix-cache#kv-cache#benchmark#hardware

Related topics