Mellum2

JetBrains’ Mellum2 open MoE code model (12B total, 2.5B active) with a 131k-token context window and tool calling.

Tool callingLong context

JetBrains’ Mellum2 open MoE code model (12B total, 2.5B active) with a 131k-token context window and tool calling.

Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

Install and sign in to Baseten

macOS or Linux

Terminal

brew tap basetenlabs/baseten
brew install baseten

Windows

Download and extract the binary, then move baseten.exe to a directory on your PATH:

Terminal

Invoke-WebRequest `
  https://github.com/basetenlabs/baseten-cli/releases/download/v0.4.0/baseten_0.4.0_windows_amd64.zip `
  -OutFile baseten.zip; Expand-Archive -Force baseten.zip .

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install the OpenAI SDK

uv pip install openai

Prefer not to install? Sign in with uvx truss login --browser and deploy with uvx truss push.

This preset serves Mellum2 12B A2.5B Instruct on a single H100 through vLLM, optimized for low-latency code generation.

Hardware

H100

Engine

vLLM 0.23.0

Context

128K

Concurrency

128

Write the config

Create and move into the project directory:

mkdir mellum2-12b-a2.5b-instruct-latency && cd mellum2-12b-a2.5b-instruct-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: model:mellum2-12b-a2.5b-instruct preset:latency

model_metadata:
  example_model_input:
    model: "JetBrains/Mellum2-12B-A2.5B-Instruct"
    messages:
      - role: user
        content: "Write a Python function to reverse a string."
    stream: true
    max_tokens: 4096
    temperature: 0.6
    top_p: 0.95
  tags:
    - openai-compatible
    - code
    - moe

base_image:
  image: vllm/vllm-openai:v0.23.0

weights:
  - source: "hf://JetBrains/Mellum2-12B-A2.5B-Instruct@main"
    mount_location: "/app/checkpoint/model"
    auth_secret_name: "hf_access_token"

resources:
  accelerator: H100
  use_gpu: true

runtime:
  predict_concurrency: 128
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 600
    stop_traffic_threshold_seconds: 120

environment_variables:
  HF_HUB_ENABLE_HF_TRANSFER: "1"
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"

secrets:
  hf_access_token: null

docker_server:
  # No --reasoning-parser: Instruct answers directly without <think> blocks.
  # Remove --enable-auto-tool-choice / --tool-call-parser if you don't need tool use.
  start_command: >-
    sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
    --tensor-parallel-size $GPU_COUNT
    --served-model-name JetBrains/Mellum2-12B-A2.5B-Instruct
    --host 0.0.0.0
    --port 8000
    --max-model-len auto
    --enable-prefix-caching
    --enable-auto-tool-choice
    --tool-call-parser hermes
    --trust-remote-code
    --load-format runai_streamer'
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000

This config runs the official vllm/vllm-openai:v0.23.0 image, the release that adds MellumForCausalLM support, and streams weights from JetBrains/Mellum2-12B-A2.5B-Instruct with the Run:ai streamer. The Hermes tool-call parser enables OpenAI-compatible function calling, and a concurrency ceiling of 128 keeps the deployment throughput-friendly for coding assistants.

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
--tensor-parallel-size	$GPU_COUNT	Number of GPUs to shard the model across.
--max-model-len	auto	Maximum context length (tokens) the server accepts per request.
--enable-prefix-caching	(no value)	Reuse KV cache across requests that share a prefix.
--enable-auto-tool-choice	(no value)	Let the model choose when to call tools without requiring tool_choice: "required".
--tool-call-parser	hermes	Server-side parser that emits structured tool_calls on the response. hermes: Hermes-style function calls.
--trust-remote-code	(no value)	Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
--load-format	runai_streamer	Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

Baseten CLI

baseten model push

You should see output similar to:

Pushing model "mellum2-12b-a2.5b-instruct-latency"...
Uploading model...
Uploaded model in 0s
✨ Model mellum2-12b-a2.5b-instruct-latency was successfully pushed ✨

  Model:       mellum2-12b-a2.5b-instruct-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

Python

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="JetBrains/Mellum2-12B-A2.5B-Instruct",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

cURL

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "JetBrains/Mellum2-12B-A2.5B-Instruct",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="JetBrains/Mellum2-12B-A2.5B-Instruct",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

Next steps

Call your model

Endpoint anatomy, authentication, and sync versus async inference

Autoscaling

Scale replicas with traffic, including scale to zero