Deploy LLMs with vLLM

Run any open-source LLM on vLLM’s serving framework.

vLLM supports a wide range of models and performance optimizations. This guide deploys a vLLM model as a custom Docker server on Baseten.

This configuration serves Qwen 2.5 3B with vLLM on an L4 GPU. The deployment process is the same for larger models like GLM-4.7. Adjust the resources and start_command to match your model’s requirements.

Set up your environment

This guide uses uvx to run Truss commands without a separate install step. Sign in to Baseten and install the OpenAI SDK. Browser login opens a tab to approve this device, so there’s no API key to copy and paste.

Sign in to Baseten

uvx truss login --browser

Install the OpenAI SDK

uv pip install openai

Hugging Face access for gated models. Some models require that you accept terms and conditions on Hugging Face before deployment. To prevent issues:

1. Accept the license for any gated models you wish to access, like Gemma 3.
2. Create a read-only user access token from your Hugging Face account.
3. Add the hf_access_token secret to your Baseten workspace.

Configure the model

Create a directory with a config.yaml file:

mkdir qwen-2-5-3b-vllm
touch qwen-2-5-3b-vllm/config.yaml

Copy the following configuration into config.yaml:

config.yaml

model_metadata:
  example_model_input:
    messages:
      - role: system
        content: "You are a helpful assistant."
      - role: user
        content: "What does Tongyi Qianwen mean?"
    stream: true
    model: Qwen/Qwen2.5-3B-Instruct
    max_tokens: 512
    temperature: 0.6
  tags:
    - openai-compatible
model_name: Qwen 2.5 3B vLLM
base_image:
  image: vllm/vllm-openai:v0.12.0
docker_server:
  start_command: vllm serve /models/qwen --served-model-name Qwen/Qwen2.5-3B-Instruct --host 0.0.0.0 --port 8000 --enable-prefix-caching
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
weights:
  - source: "hf://Qwen/Qwen2.5-3B-Instruct@aa8e72537993ba99e69dfaafa59ed015b17504d1"
    mount_location: "/models/qwen"
resources:
  accelerator: L4
  use_gpu: true
runtime:
  predict_concurrency: 256
  health_checks:
    restart_threshold_seconds: 300
    stop_traffic_threshold_seconds: 120

The base_image specifies the vLLM Docker image. The weights block uses the Baseten Delivery Network to mirror the model from Hugging Face and mount it at /models/qwen before the container starts. vLLM reads weights directly from that path and serves the model with --served-model-name, which sets the model identifier for the OpenAI-compatible API. The health_checks settings control how Baseten monitors the server after it passes the startup probe.

Deploy the model

Push the model to Baseten to start the deployment:

uvx truss push qwen-2-5-3b-vllm

You should see output like:

Deploying truss using L4:4x16 instance type.
Model Qwen 2.5 3B vLLM was successfully pushed.
View logs at https://app.baseten.co/models/XXXXXXX/logs/XXXXXXX

Copy the model URL from the output for the next step.

The first deploy can take several minutes while Baseten pulls the vLLM base image. Subsequent scale-ups reuse the cached image and start much faster.

Call the model

Call the deployed model with the OpenAI client:

call_model.py

import os
from openai import OpenAI

model_url = "https://model-XXXXXXX.api.baseten.co/environments/production/sync/v1"

client = OpenAI(
    base_url=model_url,
    api_key=os.environ.get("BASETEN_API_KEY"),
)

stream = client.chat.completions.create(
    model="Qwen/Qwen2.5-3B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What does Tongyi Qianwen mean?"}
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")

Replace the model_url with the URL from your deployment output.

Route through an external LLM gateway

To route traffic from a third-party OpenAI-compatible gateway to this deployment, see External LLM gateways. The model value the gateway sends must match the --served-model-name in the start_command above.