Voxtral

Voxtral recipes: 2 variants (Mini 4B, Mini 3B Batch), encoder-decoder architecture.

Speech-to-text

Setup

Install the Baseten CLI and sign in, then install the websockets library.

Install and sign in to Baseten

macOS or Linux

Terminal

brew tap basetenlabs/baseten
brew install baseten

Windows

Download and extract the binary, then move baseten.exe to a directory on your PATH:

Terminal

Invoke-WebRequest `
  https://github.com/basetenlabs/baseten-cli/releases/download/v0.4.0/baseten_0.4.0_windows_amd64.zip `
  -OutFile baseten.zip; Expand-Archive -Force baseten.zip .

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install websockets

uv pip install websockets

Prefer not to install? Sign in with uvx truss login --browser and deploy with uvx truss push.

Pick the model you want to deploy. Each tab is a self-contained recipe.

Mini 4B

mistralai/Voxtral-Mini-4B-Realtime-2602 is a 4B-parameter encoder-decoder model.

This preset serves Voxtral Mini Realtime on an RTX PRO 6000, tuned for low-latency streaming transcription.

Hardware

RTX_PRO_6000

Engine

vLLM (0.22.0 custom build)

Context

10K

Concurrency

48

Write the config

Create and move into the project directory:

mkdir voxtral-mini-4b-latency && cd voxtral-mini-4b-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:voxtral-mini-4b preset:latency"
model_metadata:
  repo_id: mistralai/Voxtral-Mini-4B-Realtime-2602
secrets:
  hf_access_token: null
weights:
  - source: "hf://mistralai/Voxtral-Mini-4B-Realtime-2602@main"
    mount_location: "/app/checkpoint/model"
    auth_secret_name: "hf_access_token"
environment_variables:
  VLLM_CACHE_ROOT: /cache/org/vllm
  TORCHINDUCTOR_CACHE_DIR: /cache/org/inductor
  TRITON_CACHE_DIR: /cache/org/triton
base_image:
  image: "baseten/vllm-openai:0.22.0-voxtral-realtime-fixes"
docker_server:
  start_command: >-
    sh -c "vllm serve /app/checkpoint/model
    --tensor-parallel-size 1
    --api-server-count 8
    --enable-realtime-unbounded 
    --no-enable-prefix-caching
    --realtime-reanchor-margin-tokens 1024
    --hf-overrides '{\"text_config\": {\"sliding_window\": 4096}}'
    --served-model-name mistralai/Voxtral-Mini-4B-Realtime-2602
    --host 0.0.0.0 
    --port 8000
    --max-num-seqs 48
    --max-model-len 10240
    --compilation-config '{\"cudagraph_mode\": \"PIECEWISE\", \"cudagraph_capture_sizes\": [1, 2, 4, 8, 16, 24, 32, 48], \"max_cudagraph_capture_size\": 48}'"
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/realtime
  server_port: 8000
resources:
  accelerator: RTX_PRO_6000
  cpu: "8"
  memory: 32Gi
  use_gpu: true
requirements:
  - librosa
  - pynvml
  - ffmpeg-python
  - websockets
system_packages:
  - python3.10-venv
  - ffmpeg
  - openmpi-bin
  - libopenmpi-dev
runtime:
  predict_concurrency: 48
  is_websocket_endpoint: true
  transport:
    kind: websocket
    ping_interval_seconds: null
    ping_timeout_seconds: null

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
--tensor-parallel-size	1	Number of GPUs to shard the model across.
--api-server-count	8	Number of API server processes vLLM runs in front of the engine, spreading HTTP and WebSocket handling across CPU cores.
--enable-realtime-unbounded	(no value)	Lets realtime streaming sessions run without a fixed duration cap. Available in Baseten’s patched vLLM build, not upstream vLLM.
--no-enable-prefix-caching	(no value)	Disable prefix caching, so repeated prompts do not reuse cached KV blocks.
--realtime-reanchor-margin-tokens	1024	Token margin the realtime endpoint keeps when it re-anchors a long stream’s context window, bounding memory growth over long sessions.
--hf-overrides	{"text_config": {"sliding_window": 4096}}	Overrides fields in the model’s Hugging Face config as a JSON object. The dotted form (--hf-overrides.<field>) sets the same fields one at a time.
--max-num-seqs	48	Maximum number of concurrent sequences in the batch.
--max-model-len	10240	Maximum context length (tokens) the server accepts per request.
--compilation-config	{"cudagraph_mode": "PIECEWISE", "cudagraph_capture_sizes": [1, 2, 4, 8, 16, 24, 32, 48], "max_cudagraph_capture_size": 48}	vLLM compilation passes (op fusion, dead-code elimination).

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

Baseten CLI

baseten model push

You should see output similar to:

Pushing model "voxtral-mini-4b-latency"...
Uploading model...
Uploaded model in 0s
✨ Model voxtral-mini-4b-latency was successfully pushed ✨

  Model:       voxtral-mini-4b-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

This preset exposes a WebSocket streaming endpoint at /v1/realtime for low-latency, incremental transcription. See the streaming transcription API reference for the message protocol, Python client example, and supported audio formats.

Mini 3B Batch

mistralai/Voxtral-Mini-3B-2507 is a 4.7B-parameter encoder-decoder model with up to 32K context.

This preset serves Voxtral Mini 3B on H100 40GB, tuned for batch transcription of pre-recorded audio.

Hardware

H100_40GB × 1

Engine

vLLM (0.22.0-cu129 build)

Context

32K

Concurrency

256

Write the config

Create and move into the project directory:

mkdir voxtral-mini-3b-latency && cd voxtral-mini-3b-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:voxtral-mini-3b preset:latency"
model_metadata:
  repo_id: mistralai/Voxtral-Mini-3B-2507
  tags:
    - openai-compatible
    - audio
    - asr
secrets:
  hf_access_token: null
weights:
  - source: "hf://mistralai/Voxtral-Mini-3B-2507@3060fe34b35ba5d44202ce9ff3c097642914f8f3"
    mount_location: "/app/checkpoint/model"
    auth_secret_name: "hf_access_token"
    # The HF repo ships the weights twice: mistral-format
    # consolidated.safetensors (what --load-format mistral reads) and an
    # HF-format shard pair + model.safetensors.index.json. Skip the unused
    # HF shards so the mount doesn't stream ~9GB of duplicate weights.
    ignore_patterns:
      - "model-*.safetensors"
      - "model.safetensors.index.json"
base_image:
  image: vllm/vllm-openai:v0.22.0-cu129
docker_server:
  # Voxtral ships mistral-format artifacts only (params.json + tekken.json,
  # no HF tokenizer files), so vLLM needs the mistral tokenizer/config/load
  # trio per the upstream serving recipe.
  start_command: >-
    sh -c "vllm serve /app/checkpoint/model
    --tokenizer-mode mistral
    --config-format mistral
    --load-format mistral
    --tensor-parallel-size 1
    --served-model-name mistralai/Voxtral-Mini-3B-2507
    --max-model-len 32768
    --gpu-memory-utilization 0.8
    --host 0.0.0.0
    --port 8000"
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/audio/transcriptions
  server_port: 8000
resources:
  accelerator: H100_40GB:1
  cpu: "1"
  memory: 10Gi
  use_gpu: true
requirements:
  - vllm[audio]
  - mistral-common[audio]
  - librosa
  - torch
  - torchaudio
  - pynvml
  - ffmpeg-python
system_packages:
  - python3.10-venv
  - ffmpeg
  - openmpi-bin
  - libopenmpi-dev
runtime:
  predict_concurrency: 256

Your deployment exposes the OpenAI-compatible /v1/audio/transcriptions endpoint, so the OpenAI Python SDK works with only a base_url change. Voxtral Mini 3B detects the spoken language automatically across 8 supported languages and handles audio up to about 30 minutes per request.

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
--tokenizer-mode	mistral	Selects a custom tokenizer implementation. Required for models that ship a non-standard tokenizer alongside the checkpoint.
--config-format	mistral	Model config format. Use mistral for models that ship Mistral-format params.json configs instead of HF config.json.
--load-format	mistral	Weight loading backend.
--tensor-parallel-size	1	Number of GPUs to shard the model across.
--max-model-len	32768	Maximum context length (tokens) the server accepts per request.
--gpu-memory-utilization	0.8	Fraction of GPU memory vLLM may use for weights and KV cache.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

Baseten CLI

baseten model push

You should see output similar to:

Pushing model "voxtral-mini-3b-latency"...
Uploading model...
Uploaded model in 0s
✨ Model voxtral-mini-3b-latency was successfully pushed ✨

  Model:       voxtral-mini-3b-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible chat completions API at /v1/audio/transcriptions that accepts audio inputs.

Send audio as an audio_url content item on a chat message. The model returns the transcription as the assistant message content.

Python

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="mistralai/Voxtral-Mini-3B-2507",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "audio_url",
                    "audio_url": {
                        "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"
                    },
                }
            ],
        }
    ],
)

print(response.choices[0].message.content)

cURL

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "mistralai/Voxtral-Mini-3B-2507",
    "messages": [
      {"role": "user", "content": [
        {"type": "audio_url", "audio_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"}}
      ]}
    ]
  }'

Next steps

Call your model

Endpoint anatomy, authentication, and sync versus async inference

Autoscaling

Scale replicas with traffic, including scale to zero