Qwen3.8

Dense 27B-parameter multimodal reasoning model with FP8 weights, served on a single H100 with vLLM, MTP speculative decoding, and a data-parallel vision encoder.

ReasoningTool callingMultimodal (image)Long contextAgentic

Dense 27B-parameter multimodal reasoning model with FP8 weights, served on a single H100 with vLLM, MTP speculative decoding, and a data-parallel vision encoder.

Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

Install and sign in to Baseten

macOS or Linux

Terminal

brew tap basetenlabs/baseten
brew install baseten

Windows

Download and extract the binary, then move baseten.exe to a directory on your PATH:

Terminal

Invoke-WebRequest `
  https://github.com/basetenlabs/baseten-cli/releases/download/v0.4.0/baseten_0.4.0_windows_amd64.zip `
  -OutFile baseten.zip; Expand-Archive -Force baseten.zip .

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install the OpenAI SDK

uv pip install openai

Prefer not to install? Sign in with uvx truss login --browser and deploy with uvx truss push.

This preset serves Qwen3.8-27B on H100:1 with block-scaled FP8 weights, optimized for low-latency interactive chat, reasoning, and agent workflows.

Hardware

H100 × 1

Engine

vLLM (qwen38 build)

Context

256K

Concurrency

64

Write the config

Create and move into the project directory:

mkdir qwen3.8-27b-latency && cd qwen3.8-27b-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:qwen3.8-27b preset:latency"

model_metadata:
  description: >-
    Qwen3.8-27B FP8 is a 27B dense multimodal reasoning model served through vLLM with
    OpenAI-compatible chat completions, tool calling, MTP speculative decoding, a
    data-parallel vision encoder, and CPU KV-cache offload.
  repo_id: Qwen/Qwen3.8-27B-FP8
  example_model_input:
    model: "Qwen/Qwen3.8-27B"
    messages:
      - role: user
        content: "Give me three prime numbers greater than 100."
    stream: true
    max_tokens: 512
    temperature: 1.0
    top_p: 0.95
  tags:
    - openai-compatible
    - multimodal
    - reasoning
    - tool-use

base_image:
  # qwen38 includes unreleased gated-delta-network MTP fixes required by this recipe.
  image: "vllm/vllm-openai:qwen38@sha256:4a2f33a884222f7049b983263ad9976f89452bb81affecf5b67d89ad35c1bc31"

weights:
  - source: "hf://Qwen/Qwen3.8-27B-FP8@017b9c7af6b5689d5dd426a76e0bc077eb5ca20a"
    mount_location: "/app/checkpoint/model"
    auth_secret_name: "hf_access_token"

secrets:
  hf_access_token: null

environment_variables:
  HF_HUB_ENABLE_HF_TRANSFER: "1"
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"

docker_server:
  start_command: >-
    vllm serve /app/checkpoint/model
    --served-model-name Qwen/Qwen3.8-27B
    --host 0.0.0.0
    --port 8000
    --tensor-parallel-size 1
    --max-model-len 262144
    --kv-cache-dtype fp8
    --gpu-memory-utilization 0.90
    --max-num-seqs 64
    --enable-prefix-caching
    --limit-mm-per-prompt.image 1
    --mm-encoder-tp-mode data
    --reasoning-parser qwen3
    --enable-auto-tool-choice
    --tool-call-parser qwen3_coder
    --speculative-config.method mtp
    --speculative-config.num_speculative_tokens 3
    --kv-transfer-config.kv_connector SimpleCPUOffloadConnector
    --kv-transfer-config.kv_role kv_both
    --kv-transfer-config.kv_connector_extra_config.cpu_bytes_to_use_per_rank 68719476736
    --kv-transfer-config.kv_connector_extra_config.lazy_offload false
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000

resources:
  instance_type: H100
  accelerator: H100:1
  node_count: 1
  use_gpu: true

runtime:
  predict_concurrency: 64
  health_checks:
    restart_check_delay_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120

This preset deploys the official FP8 checkpoint of Qwen3.8-27B through vLLM on a single H100. It enables MTP speculative decoding with three speculative tokens and CPU KV-cache offload, keeping time-to-first-token low while fitting the 262,144-token context window in a single GPU’s memory. The deployment exposes an OpenAI-compatible chat completions endpoint with reasoning, tool calling, and one image per prompt.

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
--tensor-parallel-size	1	Number of GPUs to shard the model across.
--max-model-len	262144	Maximum context length (tokens) the server accepts per request.
--kv-cache-dtype	fp8	KV cache numeric precision. fp8: ~2× KV cache density with negligible quality impact on most models.
--gpu-memory-utilization	0.90	Fraction of GPU memory vLLM may use for weights and KV cache.
--max-num-seqs	64	Maximum number of concurrent sequences in the batch.
--enable-prefix-caching	(no value)	Reuse KV cache across requests that share a prefix.
--limit-mm-per-prompt.image	1	Maximum number of image inputs per prompt.
--mm-encoder-tp-mode	data	Tensor-parallel mode for the multimodal vision encoder. data: Data-parallel encoding; each rank processes different images independently.
--reasoning-parser	qwen3	Server-side parser that separates reasoning output into reasoning_content. qwen3: Qwen3-family thinking format (used by Qwen3, Qwen3.5, and Qwen3.6).
--enable-auto-tool-choice	(no value)	Let the model choose when to call tools without requiring tool_choice: "required".
--tool-call-parser	qwen3_coder	Server-side parser that emits structured tool_calls on the response. qwen3_coder: Qwen3-Coder tool format.
--speculative-config.method	mtp	Speculative decoding method. mtp: Multi-token prediction head speculation.
--speculative-config.num_speculative_tokens	3	Number of tokens the draft speculator proposes per step.
--kv-transfer-config.kv_connector	SimpleCPUOffloadConnector	KV cache transfer connector for offloading cache to host memory or another node. SimpleCPUOffloadConnector: Offloads KV cache to CPU memory to extend the effective KV cache capacity beyond GPU memory.
--kv-transfer-config.kv_role	kv_both	Role this node plays in KV cache transfer. kv_both: Both sender and receiver, used for single-node CPU offload.
--kv-transfer-config.kv_connector_extra_config.cpu_bytes_to_use_per_rank	68719476736	Bytes of CPU memory allocated per rank for KV cache offload.
--kv-transfer-config.kv_connector_extra_config.lazy_offload	false	Whether to offload KV cache lazily (on demand) or eagerly. false: Eager offload; KV cache is copied to CPU memory as soon as it is produced.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

Baseten CLI

baseten model push

You should see output similar to:

Pushing model "qwen3.8-27b-latency"...
Uploading model...
Uploaded model in 0s
✨ Model qwen3.8-27b-latency was successfully pushed ✨

  Model:       qwen3.8-27b-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

Python

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.8-27B",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

cURL

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "Qwen/Qwen3.8-27B",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

To access the model’s chain of thought, enable thinking mode. The server parses the reasoning output into a separate reasoning_content field on the response:

response = client.chat.completions.create(
    model="Qwen/Qwen3.8-27B",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="Qwen/Qwen3.8-27B",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

Next steps

Call your model

Endpoint anatomy, authentication, and sync versus async inference

Autoscaling

Scale replicas with traffic, including scale to zero