Streaming

Return model output token by token as it is generated.

Token streaming returns a language model’s output incrementally as it generates tokens. The caller reads the output as it builds, so the first tokens arrive after the time to first token (TTFT) instead of after the entire response.

Baseten supports streaming across a range of inference surfaces: Model APIs (hosted, OpenAI- and Anthropic-compatible endpoints), BIS-LLM, and dedicated deployments of models packaged with Truss. Custom Docker containers that expose an OpenAI-compatible API, such as vLLM and SGLang, stream the same way.

For real-time transcription over WebSocket, see the speech quickstart. Speech endpoints use model-specific audio messages.

Use token streaming when:

• Generating the complete output takes a relatively long time.
• The first tokens are useful without the rest of the output.
• Reducing the time to first token improves the user experience.

Chat applications backed by LLMs are the clearest example.

Enable streaming

On OpenAI-compatible endpoints, set stream to true and read the response as it arrives. For custom Truss models, your model code defines the input fields and yields the response chunks.

Truss

# Self-deployed Truss model: stream from the model's predict endpoint
import os
import requests

model_id = "YOUR_MODEL_ID"

with requests.post(
    f"https://model-{model_id}.api.baseten.co/production/predict",
    headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"},
    json={"prompt": "Write a haiku about the ocean.", "stream": True},
    stream=True,
) as resp:
    for chunk in resp.iter_content():
        print(chunk.decode("utf-8"), end="", flush=True)

The OpenAI example reads raw server-sent events because the SDK’s parsed iterator hides [DONE]. The Anthropic example iterates events because text_stream hides message_stop. Both examples raise an error if the stream ends without its completion event or stop reason. Transport and API errors also propagate to the caller. Printed text is partial until these checks pass.

A completion event can still accompany an output token limit. Check the printed stop reason before accepting the answer. These examples cover Chat Completions and Messages; the Responses API uses different events.

Streaming changes when the caller sees output, not how much the model produces. The following diagram puts both delivery modes on one clock.

The top lane streams: after a short prefill, tokens fill in one at a time from the first-token mark (TTFT). The bottom lane is non-streaming: it stays empty through the same generation, then the whole response lands at once at the end. Both finish together, so the only difference is when the caller first sees output. Token timing here is illustrative, not a measured latency.

Handle incomplete responses

For Model API requests using Chat Completions or Messages, check the stream-end marker and the reason generation stopped before accepting the result. A response can reach its token limit or lose the connection after returning HTTP 200. See completion markers and recovery examples.

Set a read timeout that allows for the expected wait before output and between chunks. Use bounded retries for transient failures. Retrying starts a new generation, so keep partial output separate from the new response.