glossary/serving/slo

Service level objective

also SLO, latency target

A latency promise stated as a percentile: the first token within 500 ms for 99% of requests, or each following token within 50 ms. Serving decisions are trade-offs against these numbers, and a stack with no stated objective cannot be tuned, only made faster or slower. The objective decides the batch size, which decides throughput, which decides the bill. Chat and agent workloads want different ones.

See it happen

• explainers/
• intermediateTwo phases, two bottlenecks

  Move sequence length and batch size across a roofline and watch the limit flip from compute to memory bandwidth, the reason disaggregation exists.

Related

• glossary/
• Time to first token

  The delay between a request arriving and the first token of its answer leaving.

• Time per output token

  The average time each output token took, computed as total generation time divided by tokens generated, excluding the first.

• Goodput

  Throughput counted only for requests that met their latency targets.

• Throughput

  Tokens produced per second across all requests in flight, or requests completed per second.

• sources/
• Site Reliability Engineering, chapter 4: Service Level Objectives (Google)
• topics/
• #latency