Ling 3.0 Flash

InclusionAI Ling 3.0 Flash is a hybrid-linear sparse mixture-of-experts reasoning model with 124B total and 5.1B active parameters.

ReasoningTool callingLong context

InclusionAI Ling 3.0 Flash is a hybrid-linear sparse mixture-of-experts reasoning model with 124B total and 5.1B active parameters. This preset serves the groupwise INT4 checkpoint on two H100 GPUs through vLLM with a 256K context window, native reasoning output, and automatic tool calling.

Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

Install and sign in to Baseten

macOS or Linux

Terminal

brew tap basetenlabs/baseten
brew install baseten

Windows

Download and extract the binary, then move baseten.exe to a directory on your PATH:

Terminal

Invoke-WebRequest `
  https://github.com/basetenlabs/baseten-cli/releases/download/v0.4.0/baseten_0.4.0_windows_amd64.zip `
  -OutFile baseten.zip; Expand-Archive -Force baseten.zip .

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install the OpenAI SDK

uv pip install openai

Prefer not to install? Sign in with uvx truss login --browser and deploy with uvx truss push.

This preset serves Ling 3.0 Flash INT4 on H100:2 with groupwise INT4 weights and tensor parallel size 2, optimized for low-latency interactive reasoning and tool-calling workloads at the model’s native 256K context.

Hardware

H100 × 2

Engine

vLLM (nightly-5a4c8d99… build)

Context

256K

Concurrency

32

Write the config

Create and move into the project directory:

mkdir ling-3.0-flash-int4-latency && cd ling-3.0-flash-int4-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:ling-3.0-flash-int4 preset:latency"

model_metadata:
  description: >-
    InclusionAI Ling 3.0 Flash INT4 is a 124B-parameter, 5.1B-active hybrid-linear
    MoE reasoning model served with vLLM on two H100 GPUs. It exposes an
    OpenAI-compatible chat endpoint with a 256K context window, native reasoning
    output, and automatic tool calling.
  repo_id: inclusionAI/Ling-3.0-flash-int4
  tags:
    - openai-compatible
  example_model_input:
    model: inclusionAI/Ling-3.0-flash-int4
    messages:
      - role: user
        content: "Explain why the sky appears blue in three concise sentences."
    stream: true
    max_tokens: 512
    temperature: 0.6
    top_p: 0.95
    top_k: 20
    chat_template_kwargs:
      enable_thinking: true

base_image:
  # Ling 3 support (model, tool parser, and reasoning parser) landed after
  # vLLM v0.27.1. Pin the first available immutable nightly commit that includes it.
  image: "vllm/vllm-openai:nightly-5a4c8d99242e9e069b604d0e9b969e77f7dd501d@sha256:8bd082c274fae025b7079498fe1da65182ba1d4c2188c0f5a68c1042c38c3695"

weights:
  - source: "hf://inclusionAI/Ling-3.0-flash-int4@959d3a48cf05d2daf5fe7bdbbd3a6bb119e359f2"
    mount_location: "/app/checkpoint/model"
    auth_secret_name: "hf_access_token"

secrets:
  hf_access_token: null

environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"

docker_server:
  start_command: >-
    vllm serve /app/checkpoint/model
    --host 0.0.0.0
    --port 8000
    --served-model-name inclusionAI/Ling-3.0-flash-int4
    --max-model-len 262144
    --trust-remote-code
    --dtype bfloat16
    --gpu-memory-utilization 0.9
    --enable-chunked-prefill
    --compilation-config.cudagraph_mode FULL_AND_PIECEWISE
    --enable-prefix-caching
    --tensor-parallel-size 2
    --enable-auto-tool-choice
    --tool-call-parser ling3
    --reasoning-parser ling3
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000

resources:
  accelerator: H100:2
  use_gpu: true

runtime:
  predict_concurrency: 32
  health_checks:
    restart_check_delay_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120

This preset deploys the groupwise INT4 checkpoint of Ling 3.0 Flash through vLLM on two H100 GPUs with tensor parallelism. It enables prefix caching, chunked prefill, and full plus piecewise CUDA graphs. The deployment exposes an OpenAI-compatible chat completions endpoint with reasoning and tool calling at the model’s 256K context window.

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
--max-model-len	262144	Maximum context length (tokens) the server accepts per request.
--trust-remote-code	(no value)	Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
--dtype	bfloat16	Weight precision loaded at runtime. bfloat16: BF16 weights, no quantization.
--gpu-memory-utilization	0.9	Fraction of GPU memory vLLM may use for weights and KV cache.
--enable-chunked-prefill	(no value)	Process long prompts in chunks so decode requests keep running.
--compilation-config.cudagraph_mode	FULL_AND_PIECEWISE	CUDA graph capture mode for the compilation pass. FULL_AND_PIECEWISE: Capture CUDA graphs for both full and piecewise execution, improving decode latency across mixed batch shapes.
--enable-prefix-caching	(no value)	Reuse KV cache across requests that share a prefix.
--tensor-parallel-size	2	Number of GPUs to shard the model across.
--enable-auto-tool-choice	(no value)	Let the model choose when to call tools without requiring tool_choice: "required".
--tool-call-parser	ling3	Server-side parser that emits structured tool_calls on the response. ling3: Ling 3.0 Flash tool format.
--reasoning-parser	ling3	Server-side parser that separates reasoning output into reasoning_content. ling3: Ling 3.0 Flash thinking format.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

Baseten CLI

baseten model push

You should see output similar to:

Pushing model "ling-3.0-flash-int4-latency"...
Uploading model...
Uploaded model in 0s
✨ Model ling-3.0-flash-int4-latency was successfully pushed ✨

  Model:       ling-3.0-flash-int4-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

Python

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="inclusionAI/Ling-3.0-flash-int4",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

cURL

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "inclusionAI/Ling-3.0-flash-int4",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

The server parses the model’s chain of thought into a separate reasoning_content field on the response. Read it alongside the final answer:

response = client.chat.completions.create(
    model="inclusionAI/Ling-3.0-flash-int4",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="inclusionAI/Ling-3.0-flash-int4",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

Next steps

Call your model

Endpoint anatomy, authentication, and sync versus async inference

Autoscaling

Scale replicas with traffic, including scale to zero