Configuration reference

Configure a BIS-LLM (v2) deployment: engine selection, model weights, runtime settings, and token-based autoscaling

BIS-LLM (Baseten Inference Stack v2) deployments are configured entirely in the bis_llm block of config.yaml. There’s no engine build step: the deploy is config-only, and the settings under bis_llm.config are passed to a prebuilt serving image. Push the config with truss push.

For translating an Engine-Builder-LLM (v1) configuration to BIS-LLM, see Migrate from Engine-Builder-LLM.

Configuration structure

A BIS-LLM config.yaml has a top-level bis_llm block alongside the standard model_name, resources, and weights fields:

config.yaml

model_name: qwen2-5-7b
resources:
  accelerator: H100:1
  use_gpu: true
weights:
  - source: hf://Qwen/Qwen2.5-7B-Instruct
    mount_location: /models/qwen
bis_llm:
  version: "<version>"          # from your Baseten representative, or the platform default
  config:
    engine_backend: vllm
    checkpoint_name: Qwen/Qwen2.5-7B-Instruct
    model_name: Qwen/Qwen2.5-7B-Instruct
    tensor_parallel_size: 1
    engine_config:
      max_num_seqs: 16
      max_num_batched_tokens: 8192
      max_model_len: 32768
  additional_autoscaling_config:
    metrics:
      - name: in_flight_tokens
        target: 16000

• bis_llm.version selects the serving stack version. Omit it to use the platform default, or set the value your Baseten representative provides.
• bis_llm.config holds the engine and runtime settings covered below.
• bis_llm.additional_autoscaling_config sets token-based autoscaling. See Autoscaling BIS-LLM.
• weights mirrors model files to the Baseten Delivery Network for fast cold starts.

Engine

BIS-LLM runs your model on one of two inference engines: TensorRT-LLM or vLLM. Set engine_backend in the bis_llm.config block to choose one:

bis_llm:
  config:
    engine_backend: vllm   # or: trtllm

Baseten resolves the latest certified serving image for that engine.

engine_backend · string

The inference engine to serve with. Baseten resolves the latest promoted, multi-arch serving image that passed model-performance CI for that engine.

Options:

• trtllm: TensorRT-LLM.
• vllm: vLLM.

Required unless you set gpuTRTImage. If you set neither, the deploy fails with No serving image provided by customer, and no engine_backend specified to select a default image.

gpuTRTImage · string

A specific serving image. Takes precedence over engine_backend for image selection.

bis_llm:
  config:
    gpuTRTImage: "<image tag>"

Engine-specific runtime settings

Each engine takes its runtime settings under engine_config, using that engine’s native field names. The two vocabularies don’t overlap: TensorRT-LLM uses max_batch_size, max_num_tokens, and max_seq_len; vLLM uses max_num_seqs, max_num_batched_tokens, and max_model_len. Passing TRT-LLM names to the vLLM engine fails at startup:

ValueError: engine_config on the vLLM backend uses TRT-LLM-flavored field names
('max_batch_size' → 'max_num_seqs', 'max_num_tokens' → 'max_num_batched_tokens',
 'max_seq_len' → 'max_model_len'). Use vLLM's own names; each backend's
engine_config is native to that backend.

vLLM

bis_llm:
  config:
    engine_backend: vllm
    engine_config:
      max_num_seqs: 16
      max_num_batched_tokens: 8192
      max_model_len: 32768
      gpu_memory_utilization: 0.90
      enable_prefix_caching: true
      enable_chunked_prefill: true
      dtype: auto
      trust_remote_code: true

TensorRT-LLM

bis_llm:
  config:
    engine_backend: trtllm
    engine_config:
      backend: pytorch
      max_batch_size: 16
      max_num_tokens: 8192
      max_seq_len: 32768
      enable_chunked_prefill: true
      kv_cache_config:
        free_gpu_memory_fraction: 0.90
        enable_block_reuse: true

Model and weights

checkpoint_name · string · required

The Hugging Face repository ID (or mounted path) of the model checkpoint. BIS-LLM serves prequantized checkpoints directly, so point this at an FP8 or FP4 checkpoint when you want quantization.

model_name · string · required

The model identifier the engine loads.

served_model_name · string

The model name returned in API responses and accepted in the model field of requests.

model_path / model_path_for_tokenizer · string

Local paths to the model and its tokenizer when you mount weights with the top-level weights block. Point the tokenizer at a mounted path rather than a remote repository ID so the engine loads it locally:

weights:
  - source: hf://Qwen/Qwen2.5-7B-Instruct
    mount_location: /models/qwen
bis_llm:
  config:
    model_path: /models/qwen
    model_path_for_tokenizer: /models/qwen

Runtime settings

tensor_parallel_size · number · default: "1"

Number of GPUs for tensor parallelism. Set it to the GPU count in your accelerator.

engine_config · object

Engine runtime settings in the active engine’s native field names. See Engine-specific runtime settings.

tokenizer_limit_length · number

Maximum token length the tokenizer accepts for a request.

additional_environment_variables · object

Per-component environment variables passed into the serving pod, keyed by component (for example, Worker).

bis_llm:
  config:
    additional_environment_variables:
      Worker:
        VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"

Autoscaling

BIS-LLM scales on in-flight tokens rather than request concurrency. Set the target under additional_autoscaling_config:

bis_llm:
  additional_autoscaling_config:
    metrics:
      - name: in_flight_tokens
        target: 16000

For how token-based autoscaling behaves and how to tune the target, see Autoscaling BIS-LLM.

Advanced features

Speculative decoding, KV-aware routing, and disaggregated serving are configured through their own blocks in bis_llm.config. See Advanced features for BIS-LLM for speculative_config, b10_routing_config, and disaggregated serving. Eagle and MTP speculative decoding require Enterprise; contact your Baseten representative to enable.

Complete configuration examples

Qwen2.5-7B on vLLM

config.yaml

model_name: qwen2-5-7b
resources:
  accelerator: H100:1
  use_gpu: true
weights:
  - source: hf://Qwen/Qwen2.5-7B-Instruct
    mount_location: /models/qwen
bis_llm:
  config:
    engine_backend: vllm
    checkpoint_name: Qwen/Qwen2.5-7B-Instruct
    model_name: Qwen/Qwen2.5-7B-Instruct
    served_model_name: Qwen/Qwen2.5-7B-Instruct
    model_path: /models/qwen
    model_path_for_tokenizer: /models/qwen
    tensor_parallel_size: 1
    engine_config:
      max_num_seqs: 16
      max_num_batched_tokens: 8192
      max_model_len: 32768
      gpu_memory_utilization: 0.90
      enable_prefix_caching: true
      enable_chunked_prefill: true
      dtype: auto
      trust_remote_code: true
  additional_autoscaling_config:
    metrics:
      - name: in_flight_tokens
        target: 16000

DeepSeek V3.2 on vLLM (H200)

config.yaml

model_name: deepseek-v3-2
resources:
  accelerator: H200:4
  use_gpu: true
bis_llm:
  config:
    engine_backend: vllm
    checkpoint_name: nvidia/DeepSeek-V3.2-NVFP4
    model_name: deepseek-ai/DeepSeek-V3.2
    tensor_parallel_size: 4
    engine_config:
      max_num_seqs: 64
      max_num_batched_tokens: 16384
      max_model_len: 20480
      gpu_memory_utilization: 0.92
      enable_prefix_caching: true
      enable_chunked_prefill: true
      dtype: auto
      trust_remote_code: true
  additional_autoscaling_config:
    metrics:
      - name: in_flight_tokens
        target: 30000

Related

• BIS-LLM overview: Main engine documentation.
• Migrate from Engine-Builder-LLM: Translate a v1 configuration to BIS-LLM (v2).
• Advanced features for BIS-LLM: KV-aware routing, disaggregated serving, and speculative decoding.
• Structured outputs for BIS-LLM: JSON schema validation.
• Model deployment examples: Concrete deployment examples.