GLM-5.3 Flash

Z.ai GLM-5.3 Flash is a multimodal mixture-of-experts model with 321B total and 18B active parameters.

ReasoningTool callingMultimodal (image)Long context

Z.ai GLM-5.3 Flash is a multimodal mixture-of-experts model with 321B total and 18B active parameters. This preset serves the native FP8 checkpoint on eight H100 GPUs through vLLM with a 1M-token context window, image and video inputs, always-on reasoning, automatic tool calling, and multi-token prediction speculative decoding.

Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

Install and sign in to Baseten

macOS or Linux

Terminal

brew tap basetenlabs/baseten
brew install baseten

Windows

Download and extract the binary, then move baseten.exe to a directory on your PATH:

Terminal

Invoke-WebRequest `
  https://github.com/basetenlabs/baseten-cli/releases/download/v0.4.0/baseten_0.4.0_windows_amd64.zip `
  -OutFile baseten.zip; Expand-Archive -Force baseten.zip .

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install the OpenAI SDK

uv pip install openai

Prefer not to install? Sign in with uvx truss login --browser and deploy with uvx truss push.

This preset serves GLM-5.3 Flash on H100:8 from its native FP8 checkpoint, with a multi-token prediction head proposing five speculative tokens per step to speed up decoding.

Hardware

H100 × 8

Engine

vLLM (glm53-flash build)

Context

1M

Concurrency

16

Write the config

Create and move into the project directory:

mkdir glm-5.3-flash-latency && cd glm-5.3-flash-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:glm-5.3-flash preset:latency"

model_metadata:
  description: >-
    Z.ai GLM-5.3-Flash is a 321B-parameter, 18B-active multimodal MoE served from
    its native FP8 checkpoint with vLLM on eight H100 GPUs. It provides
    OpenAI-compatible chat completions, reasoning, tool calling, image and video
    inputs, MTP speculative decoding, and a 1M-token context window.
  repo_id: zai-org/GLM-5.3-Flash
  tags:
    - openai-compatible
    - multimodal
    - moe
    - reasoning
    - tool-use
    - fp8
  example_model_input:
    model: zai-org/GLM-5.3-Flash
    messages:
      - role: user
        content: "Explain sparse attention in three concise sentences."
    stream: true
    max_tokens: 512
    temperature: 1.0
    top_p: 0.95

base_image:
  # Dedicated image required until GLM-5.3-Flash support reaches a stable vLLM release.
  image: "vllm/vllm-openai:glm53-flash@sha256:2c6da6c6f16ed15c91e412d896dba13701f25fe1861eaec9ddaa4db34d1d21c4"

weights:
  - source: "hf://zai-org/GLM-5.3-Flash@3f1971b7b5f7a528c9c4ef6212c8785298a8c24a"
    mount_location: "/app/checkpoint/model"
    auth_secret_name: "hf_access_token"

secrets:
  hf_access_token: null

environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"

docker_server:
  start_command: >-
    vllm serve /app/checkpoint/model
    --host 0.0.0.0
    --port 8000
    --served-model-name zai-org/GLM-5.3-Flash
    --tensor-parallel-size 8
    --max-model-len 1048576
    --gpu-memory-utilization 0.90
    --max-num-seqs 16
    --enable-prefix-caching
    --limit-mm-per-prompt.image 1
    --limit-mm-per-prompt.video 1
    --tool-call-parser glm47
    --reasoning-parser glm45
    --default-chat-template-kwargs.reasoning_effort high
    --enable-auto-tool-choice
    --speculative-config.method mtp
    --speculative-config.num_speculative_tokens 5
    --no-enable-flashinfer-autotune
    --load-format runai_streamer
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000

resources:
  instance_type: H100:8

runtime:
  predict_concurrency: 16
  health_checks:
    restart_check_delay_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120

The container loads the FP8 checkpoint to /app/checkpoint/model and serves the OpenAI-compatible API on port 8000 with tensor parallel size 8 across the eight H100 GPUs. The engine caps each request at 1,048,576 tokens and runs 16 concurrent sequences, keeping weights in FP8 while the KV cache stays in BF16, because the current implementation does not support an FP8 KV cache on Hopper. Requests that omit reasoning_effort inherit the server default of high, and each prompt accepts one image and one video alongside its text.

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
--tensor-parallel-size	8	Number of GPUs to shard the model across.
--max-model-len	1048576	Maximum context length (tokens) the server accepts per request.
--gpu-memory-utilization	0.90	Fraction of GPU memory vLLM may use for weights and KV cache.
--max-num-seqs	16	Maximum number of concurrent sequences in the batch.
--enable-prefix-caching	(no value)	Reuse KV cache across requests that share a prefix.
--limit-mm-per-prompt.image	1	Maximum number of image inputs per prompt.
--limit-mm-per-prompt.video	1	Maximum number of video inputs per prompt.
--tool-call-parser	glm47	Server-side parser that emits structured tool_calls on the response. glm47: GLM-4.7-style tool format, also used by GLM-5 and GLM-5.3.
--reasoning-parser	glm45	Server-side parser that separates reasoning output into reasoning_content. glm45: GLM-4.5-style thinking format, also used by GLM-5 and GLM-5.3.
--default-chat-template-kwargs.reasoning_effort	high	Reasoning effort the chat template applies when a request omits reasoning_effort. Clients override it per request.
--enable-auto-tool-choice	(no value)	Let the model choose when to call tools without requiring tool_choice: "required".
--speculative-config.method	mtp	Speculative decoding method. mtp: Multi-token prediction head speculation.
--speculative-config.num_speculative_tokens	5	Number of tokens the draft speculator proposes per step.
--no-enable-flashinfer-autotune	(no value)	Skip the FlashInfer kernel autotuning pass that normally runs at server startup.
--load-format	runai_streamer	Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

Baseten CLI

baseten model push

You should see output similar to:

Pushing model "glm-5.3-flash-latency"...
Uploading model...
Uploaded model in 0s
✨ Model glm-5.3-flash-latency was successfully pushed ✨

  Model:       glm-5.3-flash-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

Python

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="zai-org/GLM-5.3-Flash",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

cURL

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "zai-org/GLM-5.3-Flash",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

The server parses the model’s chain of thought into a separate reasoning_content field on the response. Read it alongside the final answer:

response = client.chat.completions.create(
    model="zai-org/GLM-5.3-Flash",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="zai-org/GLM-5.3-Flash",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

Next steps

Call your model

Endpoint anatomy, authentication, and sync versus async inference

Autoscaling

Scale replicas with traffic, including scale to zero