Laguna

Laguna recipes: 2 variants (M.1, S 2.1), MoE architecture.

AgenticReasoningTool callingLong context

Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

Install and sign in to Baseten

macOS or Linux

Terminal

brew tap basetenlabs/baseten
brew install baseten

Windows

Download and extract the binary, then move baseten.exe to a directory on your PATH:

Terminal

Invoke-WebRequest `
  https://github.com/basetenlabs/baseten-cli/releases/download/v0.4.0/baseten_0.4.0_windows_amd64.zip `
  -OutFile baseten.zip; Expand-Archive -Force baseten.zip .

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install the OpenAI SDK

uv pip install openai

Prefer not to install? Sign in with uvx truss login --browser and deploy with uvx truss push.

Pick the model you want to deploy. Each tab is a self-contained recipe.

M.1

poolside/Laguna-M.1-FP8 is a MoE model with up to 256K context.

This preset serves Laguna M.1 on H100:4 with FP8 weights, optimized for low time-to-first-token on interactive reasoning and coding workloads.

Hardware

H100 × 4

Engine

vLLM 0.21.0

Context

256K

Concurrency

64

Write the config

Create and move into the project directory:

mkdir laguna-m.1-latency && cd laguna-m.1-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:laguna-m.1 preset:latency"

model_metadata:
  description: >-
    Laguna M.1 FP8 MoE reasoning model from Poolside, served with vLLM (H100 TP=4),
    OpenAI-compatible chat with tool calling and extended reasoning support.
    Latency-optimized: low max-num-seqs to minimize head-of-line blocking from long thinking traces.
  repo_id: poolside/Laguna-M.1-FP8
  trust_remote_code: true
  tags:
    - openai-compatible
    - vllm
    - moe
    - reasoning
    - agentic-coding
    - fp8
  example_model_input:
    model: poolside/laguna-m.1
    messages:
      - role: user
        content: "Write a Python retry wrapper with exponential backoff."
    stream: true
    temperature: 1.0
    top_k: 20

# ---------------------------------------------------------------------------
# Base image — vLLM with Laguna support (requires vLLM >= 0.21.0)
# ---------------------------------------------------------------------------
base_image:
  image: vllm/vllm-openai:v0.21.0
  python_executable_path: /usr/bin/python3

# ---------------------------------------------------------------------------
# Weights — FP8 quantized checkpoint (~225 GB, fits in 4× H100 / 320 GB)
# Quantization is detected automatically from the checkpoint's
# quantization_config — no extra vLLM flags needed.
# ---------------------------------------------------------------------------
weights:
  - source: "hf://poolside/Laguna-M.1-FP8@3732a656e6f0bf09f90250518d8de9c8e5c54f12"
    mount_location: "/models/laguna-m1"
    auth_secret_name: "hf_access_token"

secrets:
  hf_access_token: null

requirements:
  - runai-model-streamer==0.16.1

environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"

# ---------------------------------------------------------------------------
# Docker server — vLLM OpenAI-compatible endpoint
# ---------------------------------------------------------------------------
docker_server:
  start_command: >-
    vllm serve /models/laguna-m1
    --served-model-name poolside/laguna-m.1
    --host 0.0.0.0
    --port 8000
    --tool-call-parser poolside_v1
    --reasoning-parser poolside_v1
    --enable-auto-tool-choice
    --default-chat-template-kwargs '{"enable_thinking": true}'
    --tensor-parallel-size 4
    --max-model-len 262144
    --max-num-seqs 64
    --gpu-memory-utilization 0.95
    --trust-remote-code
    --load-format runai_streamer
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000

# ---------------------------------------------------------------------------
# Resources
# FP8 ~225 GB → 4× H100 (320 GB total VRAM) with comfortable headroom
# ---------------------------------------------------------------------------
resources:
  accelerator: H100:4
  cpu: "8"
  memory: 32Gi
  use_gpu: true

# ---------------------------------------------------------------------------
# Runtime
# ---------------------------------------------------------------------------
runtime:
  predict_concurrency: 64
  health_checks:
    restart_check_delay_seconds: 1800
    restart_threshold_seconds: 600
    stop_traffic_threshold_seconds: 180

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
--tool-call-parser	poolside_v1	Server-side parser that emits structured tool_calls on the response.
--reasoning-parser	poolside_v1	Server-side parser that separates reasoning output into reasoning_content.
--enable-auto-tool-choice	(no value)	Let the model choose when to call tools without requiring tool_choice: "required".
--default-chat-template-kwargs	{"enable_thinking": true}	Default keyword arguments applied to the chat template, used to set behaviors like enabling reasoning by default.
--tensor-parallel-size	4	Number of GPUs to shard the model across.
--max-model-len	262144	Maximum context length (tokens) the server accepts per request.
--max-num-seqs	64	Maximum number of concurrent sequences in the batch.
--gpu-memory-utilization	0.95	Fraction of GPU memory vLLM may use for weights and KV cache.
--trust-remote-code	(no value)	Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
--load-format	runai_streamer	Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

Baseten CLI

baseten model push

You should see output similar to:

Pushing model "laguna-m.1-latency"...
Uploading model...
Uploaded model in 0s
✨ Model laguna-m.1-latency was successfully pushed ✨

  Model:       laguna-m.1-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

Python

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="poolside/laguna-m.1",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

cURL

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "poolside/laguna-m.1",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

The server parses the model’s chain of thought into a separate reasoning_content field on the response. Read it alongside the final answer:

response = client.chat.completions.create(
    model="poolside/laguna-m.1",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="poolside/laguna-m.1",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

S 2.1

poolside/Laguna-S-2.1-FP8 is a MoE model with up to 256K context.

This preset serves Laguna S 2.1 on H100:8 with FP8 weights through the Baseten Inference Stack’s TensorRT-LLM engine, optimized for low latency on agentic coding and reasoning workloads.

Hardware

H100 × 8

Context

256K

Write the config

Create and move into the project directory:

mkdir laguna-s.2.1-latency && cd laguna-s.2.1-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:laguna-s.2.1 preset:latency"

model_metadata:
  description: >-
    Poolside Laguna S 2.1 FP8 agentic coding model served with TensorRT-LLM
    on eight H100 GPUs, with OpenAI-compatible chat, tool calling, reasoning,
    and a 262K-token context window.
  repo_id: poolside/Laguna-S-2.1-FP8
  trust_remote_code: true
  tags:
    - openai-compatible
    - trt-llm
    - moe
    - reasoning
    - agentic-coding
    - fp8
  example_model_input:
    model: poolside/Laguna-S-2.1-FP8
    messages:
      - role: user
        content: "Write a Python retry wrapper with exponential backoff."
    stream: true
    max_tokens: 32768
    temperature: 1.0

environment_variables: {}

resources:
  accelerator: H100:8
  use_gpu: true

bis_llm:
  version: 0.0.1-20260601190849-691c46ce
  config:
    additional_environment_variables:
      Worker:
        BAD_TOKEN_ID_SEQ_CHECK_ENABLED: "1"
        LD_LIBRARY_PATH: /usr/local/mpi/lib:/src/.venv-3.12/lib/python3.12/site-packages/tensorrt_llm/libs:/src/.venv-3.12/lib/python3.12/site-packages/tensorrt_libs:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib/python3.12/dist-packages/torch/lib:/usr/local/lib/python3.12/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
        PATH: /src/.venv-3.12/bin:/usr/local/lib/python3.12/dist-packages/torch_tensorrt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/mpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ucx/bin:/opt/amazon/efa/bin:/opt/tensorrt/bin
        PYTHONPATH: /src/.venv-3.12/lib/python3.12/site-packages:/workspace/trtllm
        PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
        TRTLLM_ENABLE_PDL: "0"
    arguments_as_json: true
    b10_autoscaling_config:
      additional_autoscaling_config:
        metrics:
          - name: in_flight_tokens
            target: 130000
      max_replica: 1
      min_replica: 1
    b10_routing_config:
      algo_selector: B10
      router_active_replicas: 1
      router_cache_miss_min_isl: 15000
      router_cache_miss_weight: 0.03
      router_decode_token_discount: 1.5
      router_disable_snapshots_in_primary: true
      router_overlap_score_weight: 6.5
      router_prefill_token_discount: 0.15
      router_queue_threshold: 2
      router_replica_sync: true
      router_snapshot_threshold: 200000
      router_temperature: 0.015
    checkpoint_name: poolside/Laguna-S-2.1-FP8
    default_sampling_params:
      max_tokens: 32768
    default_sampling_params_thinking:
      max_tokens: 32768
    default_thinking_enabled: true
    engine_config:
      backend: pytorch
      cuda_graph_config:
        batch_sizes:
          - 1
          - 2
          - 4
          - 8
        enable_padding: true
      enable_chunked_prefill: true
      guided_decoding_backend: xgrammar
      kv_cache_config:
        dtype: fp8
        enable_block_reuse: true
        enable_partial_reuse: false
        event_buffer_max_size: 16384
        free_gpu_memory_fraction: 0.95
        max_attention_window:
          - 262144
          - 512
          - 512
          - 512
      max_batch_size: 8
      max_input_len: 262144
      max_num_tokens: 16384
      max_seq_len: 262144
      moe_config:
        backend: CUTLASS
      trust_remote_code: true
    gpuTRTImage: baseten/dynamo-cache-aware-routing:maxtokclamp-20260715--trtllm-laguna-dflash-laguna-pr15666-dflash-ac9980ff16-3ee6c64a9-969251ab0@sha256:921b680c6a7c8b343ddd88ceb4099b304bcc4069d1ebd6b0f98f83651b4928c8
    model_level_stop_words:
      - </assistant>
    model_name: poolside/Laguna-S-2.1-FP8
    model_path: /models/laguna
    model_path_for_tokenizer: /models/laguna
    reasoning_parser: laguna
    served_model_name: poolside/Laguna-S-2.1-FP8
    served_model_name_response: poolside/Laguna-S-2.1-FP8
    tensor_parallel_size: 8
    tokenizer_limit_length: 262144
    tokenizer_max_new_tokens_limit: 32768
    tool_call_parser: laguna

weights:
  # Pinned to the SpinQuant-rotated FP8 checkpoint the gpuTRTImage was built
  # for. Later upstream revisions (17cacdc6 "spinquantless FP8", 9e0b8ba6
  # "RC2 1M release config") changed the quantization format and crash this
  # engine at weight load (MoE w3_w1 shape mismatch), looping until the
  # deploy times out.
  - source: hf://poolside/Laguna-S-2.1-FP8@610e62523e5ae1261d6b4d3ae7974479041d6027
    mount_location: /models/laguna
    auth_secret_name: hf_access_token

This deployment serves poolside’s Laguna S 2.1 from an FP8 checkpoint on eight H100 GPUs with tensor parallelism across all eight, running the Baseten Inference Stack’s TensorRT-LLM engine with an FP8 KV cache and a 262K-token context window. The server exposes an OpenAI-compatible chat completions API with reasoning and tool calling enabled by the model’s laguna parsers.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

Baseten CLI

baseten model push

You should see output similar to:

Pushing model "laguna-s.2.1-latency"...
Uploading model...
Uploaded model in 0s
✨ Model laguna-s.2.1-latency was successfully pushed ✨

  Model:       laguna-s.2.1-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

Python

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="poolside/Laguna-S-2.1-FP8",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

cURL

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "poolside/Laguna-S-2.1-FP8",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

Next steps

Call your model

Endpoint anatomy, authentication, and sync versus async inference

Autoscaling

Scale replicas with traffic, including scale to zero