GLM-5

Z.ai’s GLM-5 reasoning model, served from an FP8 checkpoint on B200:8.

ReasoningTool callingLong context

Z.ai’s GLM-5 reasoning model, served from an FP8 checkpoint on B200:8.

Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

Install and sign in to Baseten

macOS or Linux

Terminal

brew tap basetenlabs/baseten
brew install baseten

Windows

Download and extract the binary, then move baseten.exe to a directory on your PATH:

Terminal

Invoke-WebRequest `
  https://github.com/basetenlabs/baseten-cli/releases/download/v0.4.0/baseten_0.4.0_windows_amd64.zip `
  -OutFile baseten.zip; Expand-Archive -Force baseten.zip .

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install the OpenAI SDK

uv pip install openai

Prefer not to install? Sign in with uvx truss login --browser and deploy with uvx truss push.

This preset serves GLM-5 FP8 on B200:8, Z.ai’s frontier model tuned for the lowest time-to-first-token available.

Hardware

B200 × 8

Engine

vLLM 0.22.0

Context

128K

Concurrency

64

Write the config

Create and move into the project directory:

mkdir glm-5-latency && cd glm-5-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_metadata:
  example_model_input:
    messages:
      - role: system
        content: "You are a helpful assistant."
      - role: user
        content: "What is the meaning of life?"
    stream: true
    model: zai-org/GLM-5
    max_tokens: 32768
    temperature: 0.7
  tags:
    - openai-compatible

model_name: "model:glm-5 preset:latency"

base_image:
  image: vllm/vllm-openai:v0.22.0

docker_server:
  start_command: >-
    sh -c "VLLM_DEEP_GEMM_WARMUP=relax python3 -m vllm.entrypoints.openai.api_server
    --model /models/GLM-5-FP8
    --chat-template /models/GLM-5-FP8/chat_template.jinja
    --host 0.0.0.0 --port 8000
    --served-model-name zai-org/GLM-5
    --tensor-parallel-size 8
    --trust-remote-code
    --load-format runai_streamer
    --disable-log-stats
    --max-num-seqs 64
    --max-num-batched-tokens 8192
    --tool-call-parser glm47
    --reasoning-parser glm45
    --enable-auto-tool-choice
    --speculative-config.method mtp
    --speculative-config.num_speculative_tokens 1"
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000

weights:
  - source: "hf://zai-org/GLM-5-FP8@main"
    mount_location: "/models/GLM-5-FP8"
    ignore_patterns:
      - "*.md"
      - "*.txt"

resources:
  accelerator: B200:8
  use_gpu: true

runtime:
  predict_concurrency: 64
  health_checks:
    restart_check_delay_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
--model	/models/GLM-5-FP8	Path (or HF repo) the engine loads the model from.
--chat-template	/models/GLM-5-FP8/chat_template.jinja	Path to a Jinja chat template file that overrides the checkpoint’s default.
--tensor-parallel-size	8	Number of GPUs to shard the model across.
--trust-remote-code	(no value)	Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
--load-format	runai_streamer	Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk.
--disable-log-stats	(no value)	Suppress periodic engine stats logging.
--max-num-seqs	64	Maximum number of concurrent sequences in the batch.
--max-num-batched-tokens	8192	Maximum total tokens processed per scheduler step.
--tool-call-parser	glm47	Server-side parser that emits structured tool_calls on the response. glm47: GLM-4.7-style tool format, also used by GLM-5 and GLM-5.3.
--reasoning-parser	glm45	Server-side parser that separates reasoning output into reasoning_content. glm45: GLM-4.5-style thinking format, also used by GLM-5 and GLM-5.3.
--enable-auto-tool-choice	(no value)	Let the model choose when to call tools without requiring tool_choice: "required".
--speculative-config.method	mtp	Speculative decoding method. mtp: Multi-token prediction head speculation.
--speculative-config.num_speculative_tokens	1	Number of tokens the draft speculator proposes per step.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

Baseten CLI

baseten model push

You should see output similar to:

Pushing model "glm-5-latency"...
Uploading model...
Uploaded model in 0s
✨ Model glm-5-latency was successfully pushed ✨

  Model:       glm-5-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

Python

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="zai-org/GLM-5",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

cURL

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "zai-org/GLM-5",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

The server parses the model’s chain of thought into a separate reasoning_content field on the response. Read it alongside the final answer:

response = client.chat.completions.create(
    model="zai-org/GLM-5",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="zai-org/GLM-5",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

Next steps

Call your model

Endpoint anatomy, authentication, and sync versus async inference

Autoscaling

Scale replicas with traffic, including scale to zero