Streaming
Stream chat completions as Server-Sent Events with stream:true — the SSE format, the [DONE] sentinel, and usage in the final chunk.
Set "stream": true on a chat completion to
receive the response token-by-token as Server-Sent Events (SSE) instead of
waiting for the whole completion.
The SSE format
The response is sent with Content-Type: text/event-stream. Each event is a
single data: line whose payload is a JSON chat.completion.chunk object,
separated from the next by a blank line:
data: {"id":"chatcmpl-…","object":"chat.completion.chunk","model":"openrelay/gpt-oss-120b","choices":[{"index":0,"delta":{"role":"assistant"}}]}
data: {"id":"chatcmpl-…","object":"chat.completion.chunk","model":"openrelay/gpt-oss-120b","choices":[{"index":0,"delta":{"content":"Hello"}}]}
data: {"id":"chatcmpl-…","object":"chat.completion.chunk","model":"openrelay/gpt-oss-120b","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":"stop"}]}Concatenate the choices[0].delta.content fragments across chunks to assemble
the full message. As with unary responses, the model field on every chunk is
the public OpenRelay id you requested.
Usage in the final chunk
The stream always ends with a final chunk that carries the request's usage
totals (and typically an empty choices array), so you can account for token
usage without a second request:
data: {"id":"chatcmpl-…","object":"chat.completion.chunk","model":"openrelay/gpt-oss-120b","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":9,"total_tokens":21}}Usage is always reported on a streamed response — the gateway requests it
on your behalf, so you don't need to set stream_options.include_usage
yourself.
The [DONE] sentinel
After the final chunk, the stream emits the OpenAI-style termination sentinel and closes:
data: [DONE]Stop reading when you receive [DONE].
Consuming the stream
curl -N https://inference.openrelay.inc/v1/chat/completions \
-H "Authorization: Bearer $OPENRELAY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "openrelay/gpt-oss-120b",
"stream": true,
"messages": [{ "role": "user", "content": "Tell me a joke." }]
}'-N disables curl's buffering so events print as they arrive.
from openai import OpenAI
client = OpenAI(
base_url="https://inference.openrelay.inc/v1",
api_key="vl_your_api_key",
)
stream = client.chat.completions.create(
model="openrelay/gpt-oss-120b",
messages=[{"role": "user", "content": "Tell me a joke."}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content if chunk.choices else None
if delta:
print(delta, end="", flush=True)import OpenAI from 'openai';
const client = new OpenAI({
baseURL: 'https://inference.openrelay.inc/v1',
apiKey: 'vl_your_api_key',
});
const stream = await client.chat.completions.create({
model: 'openrelay/gpt-oss-120b',
messages: [{ role: 'user', content: 'Tell me a joke.' }],
stream: true,
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content ?? '');
}The OpenAI SDKs parse the SSE stream and the [DONE] sentinel for you. See
Using OpenAI SDKs for full setup.