Nemotron 3.5

NVIDIA’s Nemotron 3.5 Lightning is a hybrid Mamba-2, MoE, and attention reasoning model with 30B total and 3B active parameters.

ReasoningTool callingLong context

NVIDIA’s Nemotron 3.5 Lightning is a hybrid Mamba-2, MoE, and attention reasoning model with 30B total and 3B active parameters. This preset serves the NVFP4 checkpoint on one H100 with vLLM and DFlash speculative decoding, targeting high throughput at the model’s native 1M-token context.

Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

Install and sign in to Baseten

macOS or Linux

Terminal

brew tap basetenlabs/baseten
brew install baseten

Windows

Download and extract the binary, then move baseten.exe to a directory on your PATH:

Terminal

Invoke-WebRequest `
  https://github.com/basetenlabs/baseten-cli/releases/download/v0.4.0/baseten_0.4.0_windows_amd64.zip `
  -OutFile baseten.zip; Expand-Archive -Force baseten.zip .

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install the OpenAI SDK

uv pip install openai

Prefer not to install? Sign in with uvx truss login --browser and deploy with uvx truss push.

This preset serves Nemotron 3.5 Lightning 30B-A3B on one H100 under vLLM with the NVFP4 checkpoint and DFlash speculative decoding, optimized for total token throughput at the model’s native 1M-token context window.

Hardware

H100

Engine

vLLM (0.27.0-x86_64 build)

Context

1M

Concurrency

512

Write the config

Create and move into the project directory:

mkdir nemotron-3.5-lightning-throughput && cd nemotron-3.5-lightning-throughput

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:nemotron-3.5-lightning preset:throughput"

model_metadata:
  description: >-
    NVIDIA Nemotron 3.5 Lightning 30B-A3B (NVFP4) on a single H100 with vLLM
    v0.27.0 and DFlash speculative decoding (K=3), tuned for the checkpoint's
    native 1,048,576-token context window. The verifier and drafter are pinned
    public NVIDIA Hugging Face repositories mounted through BDN.
  repo_id: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
  example_model_input:
    model: nvidia/nemotron-3.5-lightning-nvfp4
    messages:
      - role: user
        content: "Reply with exactly: deployment healthy"
    max_tokens: 32
    temperature: 0
    stream: false
  tags:
    - openai-compatible
    - vllm
    - dflash
    - nemotron
    - nvfp4
    - h100
    - throughput
    - tool-calling
    - 1m-context
    - public-huggingface

base_image:
  image: vllm/vllm-openai:v0.27.0-x86_64

weights:
  - source: "hf://nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4@0dcd680e5585c791728c83342b311d0a0026dbeb"
    mount_location: /models/nemotron-3.5-lightning-nvfp4
    auth_secret_name: hf_access_token
    allow_patterns:
      - "*.safetensors"
      - "*.json"
      - "*.py"
      - "*.jinja"
      - "*.txt"
      - "*.md"
  - source: "hf://nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash@7fc1f1ff4b82b917efbd0710df0872c2bb89caa5"
    mount_location: /models/nemotron-3.5-lightning-nvfp4-dflash
    auth_secret_name: hf_access_token
    allow_patterns:
      - "*.safetensors"
      - "*.json"
      - "*.md"

secrets:
  hf_access_token: null

docker_server:
  start_command: >-
    vllm serve /models/nemotron-3.5-lightning-nvfp4
    --served-model-name nvidia/nemotron-3.5-lightning-nvfp4
    --host 0.0.0.0
    --port 8000
    --trust-remote-code
    --reasoning-parser nemotron_v3
    --enable-auto-tool-choice
    --tool-call-parser qwen3_coder
    --max-num-seqs 512
    --max-model-len 1048576
    --max-num-batched-tokens 32768
    --enable-prefix-caching
    --mamba-cache-mode align
    --quantization modelopt_fp4
    --speculative-config '{"method":"dflash","model":"/models/nemotron-3.5-lightning-nvfp4-dflash","num_speculative_tokens":3}'
    --moe-backend humming
    --linear-backend humming
    --mamba-backend flashinfer
    --mamba-ssm-cache-dtype float16
    --enable-mamba-cache-stochastic-rounding
    --mamba-cache-philox-rounds 5
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000

resources:
  accelerator: H100
  use_gpu: true

runtime:
  predict_concurrency: 512
  streaming_read_timeout: 3600
  health_checks:
    startup_threshold_seconds: 1800
    restart_check_delay_seconds: 300
    restart_threshold_seconds: 900
    stop_traffic_threshold_seconds: 120

environment_variables:
  VLLM_USE_RUST_FRONTEND: "1"
  VLLM_WORKER_MULTIPROC_METHOD: spawn
  VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS: "0"
  VLLM_CACHE_ROOT: /tmp/vllmcache
  TORCHINDUCTOR_CACHE_DIR: /tmp/inductor
  HF_HUB_OFFLINE: "1"

This config mounts NVIDIA’s public NVFP4 verifier and DFlash drafter checkpoints through BDN, then launches vLLM with Humming MoE and quantized linear kernels, FlashInfer Mamba, and FP16 Mamba state with stochastic rounding. DFlash speculative decoding with three draft tokens combines with aligned prefix caching and a 512-sequence batch ceiling to push throughput on the model’s full 1,048,576-token context.

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
--trust-remote-code	(no value)	Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
--reasoning-parser	nemotron_v3	Server-side parser that separates reasoning output into reasoning_content. nemotron_v3: Nemotron 3.x thinking format.
--enable-auto-tool-choice	(no value)	Let the model choose when to call tools without requiring tool_choice: "required".
--tool-call-parser	qwen3_coder	Server-side parser that emits structured tool_calls on the response. qwen3_coder: Qwen3-Coder tool format.
--max-num-seqs	512	Maximum number of concurrent sequences in the batch.
--max-model-len	1048576	Maximum context length (tokens) the server accepts per request.
--max-num-batched-tokens	32768	Maximum total tokens processed per scheduler step.
--enable-prefix-caching	(no value)	Reuse KV cache across requests that share a prefix.
--mamba-cache-mode	align	Layout strategy for the Mamba state-space model cache on hybrid architectures. align: Align cache blocks across attention and Mamba layers for shared prefix reuse.
--quantization	modelopt_fp4	Weight quantization method loaded at runtime. modelopt_fp4: NVIDIA Model Optimizer NVFP4 (4-bit float) weights with W4A16 dequantization on Hopper.
--speculative-config	{"method":"dflash","model":"/models/nemotron-3.5-lightning-nvfp4-dflash","num_speculative_tokens":3}	Speculative decoding configuration as a JSON object. The dotted form (--speculative-config.method, --speculative-config.num_speculative_tokens, …) sets the same fields one at a time.
--moe-backend	humming	MoE expert dispatch kernel. Engine-specific values select between routing implementations tuned for different hardware or model layouts.
--linear-backend	humming	Kernel vLLM uses for linear (non-MoE) layers. humming: NVIDIA Humming quantized linear kernels for NVFP4 inference.
--mamba-backend	flashinfer	Kernel vLLM uses for Mamba-2 state-space layers. flashinfer: FlashInfer Mamba kernels, optimized for hybrid attention-SSM architectures.
--mamba-ssm-cache-dtype	float16	Precision for the Mamba state-space model cache on hybrid attention and SSM architectures.
--enable-mamba-cache-stochastic-rounding	(no value)	Use stochastic rounding for FP16 Mamba state updates, preventing bias accumulation in the recurrent cache.
--mamba-cache-philox-rounds	5	Number of Philox counter rounds per stochastic rounding step in the Mamba state cache.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

Baseten CLI

baseten model push

You should see output similar to:

Pushing model "nemotron-3.5-lightning-throughput"...
Uploading model...
Uploaded model in 0s
✨ Model nemotron-3.5-lightning-throughput was successfully pushed ✨

  Model:       nemotron-3.5-lightning-throughput (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

Python

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="nvidia/nemotron-3.5-lightning-nvfp4",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

cURL

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "nvidia/nemotron-3.5-lightning-nvfp4",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

The server parses the model’s chain of thought into a separate reasoning_content field on the response. Read it alongside the final answer:

response = client.chat.completions.create(
    model="nvidia/nemotron-3.5-lightning-nvfp4",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="nvidia/nemotron-3.5-lightning-nvfp4",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

Next steps

Call your model

Endpoint anatomy, authentication, and sync versus async inference

Autoscaling

Scale replicas with traffic, including scale to zero