glossary/tooling/openai-compatible-api
OpenAI-compatible API
also chat completions API
The HTTP interface every serving engine exposes: chat completions with a list of messages, streaming over server-sent events, the same request and response shapes OpenAI defined. It is the reason a client can switch between vLLM, SGLang, TensorRT-LLM and a hosted provider by changing a base URL. It is also where the serving metrics are measured from: time to first token is the first streamed chunk, seen from the client.
one base URL
Changed, to move a client between engines and providers.
Related
- glossary/
- Time to first token
The delay between a request arriving and the first token of its answer leaving.
- vLLM
The open-source serving engine from UC Berkeley that introduced PagedAttention and made continuous batching with paged KV cache the default architecture.
- SGLang
A serving engine and frontend language built around RadixAttention, its radix-tree prefix cache, and a compressed finite-state machine for fast constrained decoding.
- sources/
- topics/