> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Configuration reference > Configure a BIS-LLM (v2) deployment: engine selection, model weights, runtime settings, and token-based autoscaling BIS-LLM (Baseten Inference Stack v2) deployments are configured entirely in the `bis_llm` block of `config.yaml`. There's no engine build step: the deploy is config-only, and the settings under `bis_llm.config` are passed to a prebuilt serving image. Push the config with `truss push`. For translating an Engine-Builder-LLM (v1) configuration to BIS-LLM, see [Migrate from Engine-Builder-LLM](/engines/bis-llm/migrate-from-v1). ## Configuration structure A BIS-LLM `config.yaml` has a top-level `bis_llm` block alongside the standard `model_name`, `resources`, and `weights` fields: ```yaml config.yaml theme={"system"} model_name: qwen2-5-7b resources: accelerator: H100:1 use_gpu: true weights: - source: hf://Qwen/Qwen2.5-7B-Instruct mount_location: /models/qwen bis_llm: version: "" # from your Baseten representative, or the platform default config: engine_backend: vllm checkpoint_name: Qwen/Qwen2.5-7B-Instruct model_name: Qwen/Qwen2.5-7B-Instruct tensor_parallel_size: 1 engine_config: max_num_seqs: 16 max_num_batched_tokens: 8192 max_model_len: 32768 additional_autoscaling_config: metrics: - name: in_flight_tokens target: 16000 ``` * `bis_llm.version` selects the serving stack version. Omit it to use the platform default, or set the value your Baseten representative provides. * `bis_llm.config` holds the engine and runtime settings covered below. * `bis_llm.additional_autoscaling_config` sets token-based autoscaling. See [Autoscaling BIS-LLM](/engines/performance-concepts/autoscaling-engines#bis-llm). * `weights` mirrors model files to the [Baseten Delivery Network](/development/model/bdn) for fast cold starts. ## Engine BIS-LLM runs your model on one of two inference engines: TensorRT-LLM or vLLM. Set `engine_backend` in the `bis_llm.config` block to choose one: ```yaml theme={"system"} bis_llm: config: engine_backend: vllm # or: trtllm ``` Baseten resolves the latest certified serving image for that engine. The inference engine to serve with. Baseten resolves the latest promoted, multi-arch serving image that passed model-performance CI for that engine. **Options:** * `trtllm`: TensorRT-LLM. * `vllm`: vLLM. Required unless you set `gpuTRTImage`. If you set neither, the deploy fails with `No serving image provided by customer, and no engine_backend specified to select a default image.` A specific serving image. Takes precedence over `engine_backend` for image selection. ```yaml theme={"system"} bis_llm: config: gpuTRTImage: "" ``` ### Engine-specific runtime settings Each engine takes its runtime settings under `engine_config`, using that engine's native field names. The two vocabularies don't overlap: TensorRT-LLM uses `max_batch_size`, `max_num_tokens`, and `max_seq_len`; vLLM uses `max_num_seqs`, `max_num_batched_tokens`, and `max_model_len`. Passing TRT-LLM names to the vLLM engine fails at startup: ```output theme={"system"} ValueError: engine_config on the vLLM backend uses TRT-LLM-flavored field names ('max_batch_size' → 'max_num_seqs', 'max_num_tokens' → 'max_num_batched_tokens', 'max_seq_len' → 'max_model_len'). Use vLLM's own names; each backend's engine_config is native to that backend. ``` ```yaml theme={"system"} bis_llm: config: engine_backend: vllm engine_config: max_num_seqs: 16 max_num_batched_tokens: 8192 max_model_len: 32768 gpu_memory_utilization: 0.90 enable_prefix_caching: true enable_chunked_prefill: true dtype: auto trust_remote_code: true ``` ```yaml theme={"system"} bis_llm: config: engine_backend: trtllm engine_config: backend: pytorch max_batch_size: 16 max_num_tokens: 8192 max_seq_len: 32768 enable_chunked_prefill: true kv_cache_config: free_gpu_memory_fraction: 0.90 enable_block_reuse: true ``` ## Model and weights The Hugging Face repository ID (or mounted path) of the model checkpoint. BIS-LLM serves prequantized checkpoints directly, so point this at an FP8 or FP4 checkpoint when you want quantization. The model identifier the engine loads. The model name returned in API responses and accepted in the `model` field of requests. Local paths to the model and its tokenizer when you mount weights with the top-level `weights` block. Point the tokenizer at a mounted path rather than a remote repository ID so the engine loads it locally: ```yaml theme={"system"} weights: - source: hf://Qwen/Qwen2.5-7B-Instruct mount_location: /models/qwen bis_llm: config: model_path: /models/qwen model_path_for_tokenizer: /models/qwen ``` ## Runtime settings Number of GPUs for tensor parallelism. Set it to the GPU count in your `accelerator`. Engine runtime settings in the active engine's native field names. See [Engine-specific runtime settings](#engine-specific-runtime-settings). Maximum token length the tokenizer accepts for a request. Per-component environment variables passed into the serving pod, keyed by component (for example, `Worker`). ```yaml theme={"system"} bis_llm: config: additional_environment_variables: Worker: VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1" ``` ## Autoscaling BIS-LLM scales on in-flight tokens rather than request concurrency. Set the target under `additional_autoscaling_config`: ```yaml theme={"system"} bis_llm: additional_autoscaling_config: metrics: - name: in_flight_tokens target: 16000 ``` For how token-based autoscaling behaves and how to tune the target, see [Autoscaling BIS-LLM](/engines/performance-concepts/autoscaling-engines#bis-llm). ## Advanced features Speculative decoding, KV-aware routing, and disaggregated serving are configured through their own blocks in `bis_llm.config`. See [Advanced features for BIS-LLM](/engines/bis-llm/advanced-features) for `speculative_config`, `b10_routing_config`, and disaggregated serving. Eagle and MTP speculative decoding require Enterprise; [contact your Baseten representative](mailto:support@baseten.co) to enable. ## Complete configuration examples ### Qwen2.5-7B on vLLM ```yaml config.yaml theme={"system"} model_name: qwen2-5-7b resources: accelerator: H100:1 use_gpu: true weights: - source: hf://Qwen/Qwen2.5-7B-Instruct mount_location: /models/qwen bis_llm: config: engine_backend: vllm checkpoint_name: Qwen/Qwen2.5-7B-Instruct model_name: Qwen/Qwen2.5-7B-Instruct served_model_name: Qwen/Qwen2.5-7B-Instruct model_path: /models/qwen model_path_for_tokenizer: /models/qwen tensor_parallel_size: 1 engine_config: max_num_seqs: 16 max_num_batched_tokens: 8192 max_model_len: 32768 gpu_memory_utilization: 0.90 enable_prefix_caching: true enable_chunked_prefill: true dtype: auto trust_remote_code: true additional_autoscaling_config: metrics: - name: in_flight_tokens target: 16000 ``` ### DeepSeek V3.2 on vLLM (H200) ```yaml config.yaml theme={"system"} model_name: deepseek-v3-2 resources: accelerator: H200:4 use_gpu: true bis_llm: config: engine_backend: vllm checkpoint_name: nvidia/DeepSeek-V3.2-NVFP4 model_name: deepseek-ai/DeepSeek-V3.2 tensor_parallel_size: 4 engine_config: max_num_seqs: 64 max_num_batched_tokens: 16384 max_model_len: 20480 gpu_memory_utilization: 0.92 enable_prefix_caching: true enable_chunked_prefill: true dtype: auto trust_remote_code: true additional_autoscaling_config: metrics: - name: in_flight_tokens target: 30000 ``` ## Related * [BIS-LLM overview](/engines/bis-llm/overview): Main engine documentation. * [Migrate from Engine-Builder-LLM](/engines/bis-llm/migrate-from-v1): Translate a v1 configuration to BIS-LLM (v2). * [Advanced features for BIS-LLM](/engines/bis-llm/advanced-features): KV-aware routing, disaggregated serving, and speculative decoding. * [Structured outputs for BIS-LLM](/inference/structured-outputs): JSON schema validation. * [Model deployment examples](/examples/overview): Concrete deployment examples.