Nemotron 3

Nemotron 3 recipes: 2 variants (Nano 30B A3B, Super 120B A12B), Hybrid MoE and MoE architectures.

ReasoningTool callingLong contextAgentic

Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

Install and sign in to Baseten

macOS or Linux

Terminal

brew tap basetenlabs/baseten
brew install baseten

Windows

Download and extract the binary, then move baseten.exe to a directory on your PATH:

Terminal

Invoke-WebRequest `
  https://github.com/basetenlabs/baseten-cli/releases/download/v0.4.0/baseten_0.4.0_windows_amd64.zip `
  -OutFile baseten.zip; Expand-Archive -Force baseten.zip .

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install the OpenAI SDK

uv pip install openai

Prefer not to install? Sign in with uvx truss login --browser and deploy with uvx truss push.

Pick the model you want to deploy. Each tab is a self-contained recipe.

Nano 30B A3B

nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 is a 30B-parameter hybrid MoE model (3.5B active per token) with up to 256K context.

This preset serves NVIDIA Nemotron 3 Nano 30B A3B on one H100 with FP8 weights and KV cache, optimized for low-latency reasoning at the 32K context window sized for the NVIDIA cascaded voice pipeline.

Hardware

H100

Engine

vLLM (0.22.0-cu129 build)

Context

32K

Concurrency

8

Write the config

Create and move into the project directory:

mkdir nemotron-3-nano-latency && cd nemotron-3-nano-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:nemotron-3-nano preset:latency"
model_metadata:
  description: >-
    NVIDIA Nemotron 3 Nano 30B A3B (text-only) reasoning model served via vLLM on a single
    H100. A Mamba2-Transformer hybrid MoE — 23 Mamba-2 layers, 23 MoE layers and 6 GQA
    attention layers, 128 routed experts plus 1 shared expert, 6 active per token, 3.5B
    active of 30B total parameters. FP8 checkpoint (weights and KV cache), with tool
    calling and a toggleable reasoning trace.
  repo_id: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8
  example_model_input:
    model: nvidia/nemotron-3-nano
    messages:
      - role: user
        content: "What is the capital of France?"
    stream: true
    max_tokens: 512
    temperature: 0.2
    extra_body:
      chat_template_kwargs:
        enable_thinking: false
  tags:
    - openai-compatible
base_image:
  # Matches the sibling NVIDIA vLLM entries. Needs --reasoning-parser-plugin,
  # present since v0.20.0.
  image: vllm/vllm-openai:v0.22.0-cu129
weights:
  # Public, ungated upstream repo — not a baseten-admin mirror. The Xet 403 that
  # this repo is known for is specific to the model_cache: path; weights: pulls it
  # fine (33 GB in 34s on first deploy).
  - source: "hf://nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8@f8dc1c0afee92f44417695b4f5ddca9afc95ea58"
    mount_location: "/app/checkpoint/model"
    auth_secret_name: "hf_access_token"
secrets:
  hf_access_token: null
docker_server:
  # The reasoning parser is nano_v3, shipped inside the HF repo and loaded as a
  # plugin — NOT the built-in nemotron_v3 that the omni entry and the NIM use.
  # weights: mounts the whole repo, so the plugin is already on disk.
  #
  # --kv-cache-dtype fp8: the checkpoint quantized the KV cache to FP8 with the
  #   weights (attention and the Mamba layers feeding it stay BF16).
  # --mamba-ssm-cache-dtype float32: keeps the hybrid Mamba SSM state cache in
  #   fp32 to avoid accuracy drift. Beyond the model card; matches omni.
  # --max-model-len 32768 / --max-num-seqs 8: latency preset, sized for the
  #   cascaded voice pipeline. A throughput preset would raise both.
  # --tensor-parallel-size 1: single-GPU preset, so it needs no `sh -c` wrapper
  #   at all. The multi-GPU entries derive TP from nvidia-smi and must use
  #   SINGLE quotes (see #283) to defer expansion into the container; invoking
  #   vllm directly avoids that whole class of quoting bug. Keep it that way.
  start_command: >-
    vllm serve /app/checkpoint/model
    --tensor-parallel-size 1
    --host 0.0.0.0
    --port 8000
    --served-model-name nvidia/nemotron-3-nano
    --trust-remote-code
    --max-model-len 32768
    --max-num-seqs 8
    --gpu-memory-utilization 0.9
    --kv-cache-dtype fp8
    --mamba-ssm-cache-dtype float32
    --reasoning-parser-plugin /app/checkpoint/model/nano_v3_reasoning_parser.py
    --reasoning-parser nano_v3
    --enable-auto-tool-choice
    --tool-call-parser qwen3_coder
    --enable-prefix-caching
    --load-format runai_streamer
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"
  # Model card: FlashInfer FP8 MoE kernels for this checkpoint.
  VLLM_USE_FLASHINFER_MOE_FP8: "1"
resources:
  accelerator: H100
  use_gpu: true
runtime:
  predict_concurrency: 8
  health_checks:
    restart_check_delay_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120

This preset deploys the FP8 checkpoint of Nemotron 3 Nano 30B A3B through vLLM on a single H100 with the model’s own nano_v3 reasoning parser loaded as a plugin. The KV cache is FP8 to match the checkpoint, and the Mamba SSM state cache stays in FP32 to avoid accuracy drift. The deployment exposes an OpenAI-compatible chat completions endpoint with reasoning and tool calling at a 32K context window.

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
--tensor-parallel-size	1	Number of GPUs to shard the model across.
--trust-remote-code	(no value)	Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
--max-model-len	32768	Maximum context length (tokens) the server accepts per request.
--max-num-seqs	8	Maximum number of concurrent sequences in the batch.
--gpu-memory-utilization	0.9	Fraction of GPU memory vLLM may use for weights and KV cache.
--kv-cache-dtype	fp8	KV cache numeric precision. fp8: ~2× KV cache density with negligible quality impact on most models.
--mamba-ssm-cache-dtype	float32	Precision for the Mamba state-space model cache on hybrid attention and SSM architectures.
--reasoning-parser-plugin	/app/checkpoint/model/nano_v3_reasoning_parser.py	Path to a custom reasoning parser loaded as a vLLM plugin, used when the parser is not built into vLLM.
--reasoning-parser	nano_v3	Server-side parser that separates reasoning output into reasoning_content. nano_v3: Nemotron 3 Nano thinking format (shipped inside the HF repo as a plugin).
--enable-auto-tool-choice	(no value)	Let the model choose when to call tools without requiring tool_choice: "required".
--tool-call-parser	qwen3_coder	Server-side parser that emits structured tool_calls on the response. qwen3_coder: Qwen3-Coder tool format.
--enable-prefix-caching	(no value)	Reuse KV cache across requests that share a prefix.
--load-format	runai_streamer	Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

Baseten CLI

baseten model push

You should see output similar to:

Pushing model "nemotron-3-nano-latency"...
Uploading model...
Uploaded model in 0s
✨ Model nemotron-3-nano-latency was successfully pushed ✨

  Model:       nemotron-3-nano-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

Python

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="nvidia/nemotron-3-nano",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

cURL

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "nvidia/nemotron-3-nano",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

The server parses the model’s chain of thought into a separate reasoning_content field on the response. Read it alongside the final answer:

response = client.chat.completions.create(
    model="nvidia/nemotron-3-nano",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="nvidia/nemotron-3-nano",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

Super 120B A12B

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 is a 120B-parameter MoE model with 12B active per token.

This preset serves Nemotron 3 Super 120B A12B on B200:4 through Baseten Inference Stack (TensorRT-LLM) with NVFP4 weights, expert parallelism, and MTP speculative decoding. It targets high-throughput reasoning.

Hardware

B200 × 4

Engine

TRT-LLM v2

Context

128K

Concurrency

32

Write the config

Create and move into the project directory:

mkdir nemotron-3-super-120b-a12b-throughput && cd nemotron-3-super-120b-a12b-throughput

Then create a file named config.yaml and paste the following:

config.yaml

model_name: model:nemotron-3-super-120b-a12b preset:throughput

model_metadata:
  example_model_input:
    model: "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4"
    max_tokens: 512
    messages:
      - role: user
        content: Tell me everything you know about optimized inference.
    stream: true
    temperature: 0.5
  tags:
    - openai-compatible

resources:
  accelerator: B200:4
  cpu: "1"
  memory: 10Gi
  use_gpu: true

environment_variables:
  PYTORCH_CUDA_ALLOC_CONF: "expandable_segments:True"
  TRTLLM_ENABLE_PDL: "1"
  BAD_TOKEN_ID_SEQ_CHECK_ENABLED: "1"
  ENABLE_B10_LOOKAHEAD: "0"

secrets:
  hf_access_token: null

trt_llm:
  inference_stack: v2
  build:
    checkpoint_repository:
      repo: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
      revision: main
      source: HF
      runtime_secret_name: hf_access_token
  runtime:
    enable_chunked_prefill: true
    max_batch_size: 32
    max_num_tokens: 16384
    max_seq_len: 131072
    tensor_parallel_size: 4
    served_model_name: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
    patch_kwargs:
      reasoning_parser: nemotron3
      tool_call_parser: qwen3_coder
      tokenizer_limit_length: 131072
      arguments_as_json: true
      engine_config:
        backend: pytorch
        enable_chunked_prefill: true
        enable_iter_perf_stats: true
        max_batch_size: 32
        max_beam_width: 1
        max_input_len: 131072
        max_num_tokens: 16384
        max_seq_len: 131072
        trust_remote_code: true
        moe_expert_parallel_size: 4
        cuda_graph_config:
          enable_padding: true
          max_batch_size: 32
        kv_cache_config:
          dtype: fp8
          enable_block_reuse: false
          free_gpu_memory_fraction: 0.8
          mamba_ssm_cache_dtype: float32
        moe_config:
          backend: TRTLLM
        speculative_config:
          decoding_type: MTP
          num_nextn_predict_layers: 3
          allow_advanced_sampling: true

This config tells Baseten to compile a TensorRT-LLM engine for Nemotron 3 Super 120B A12B on four B200 GPUs with NVFP4-quantized weights, wiring the Qwen3-coder tool-call parser and Nemotron 3 reasoning parser into the engine config. Expert parallelism across all four GPUs, MTP speculative decoding with three draft tokens, and chunked prefill combine to push high reasoning throughput, serving up to a 128K context window.

Key parameters

Baseten Inference Stack (BIS) reads these fields from the trt_llm block. Each one shapes how the engine is built and served:

Parameter	Value
Tensor parallel size	4
Max sequence length	131072
Max batch size	32
Max batched tokens	16384
Chunked prefill	enabled
Inference stack	v2
Served model name	nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

Baseten CLI

baseten model push

You should see output similar to:

Pushing model "nemotron-3-super-120b-a12b-throughput"...
Uploading model...
Uploaded model in 0s
✨ Model nemotron-3-super-120b-a12b-throughput was successfully pushed ✨

  Model:       nemotron-3-super-120b-a12b-throughput (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

Python

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

cURL

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

Next steps

Call your model

Endpoint anatomy, authentication, and sync versus async inference

Autoscaling

Scale replicas with traffic, including scale to zero