> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# DeepSeek V4

> DeepSeek V4 recipes: 2 variants (Flash, Pro), MoE architecture.

[Reasoning](https://docs.baseten.co/examples/models/capabilities/reasoning)
[Tool calling](https://docs.baseten.co/examples/models/capabilities/tool-calling)
[Agentic](https://docs.baseten.co/examples/models/capabilities/agentic)
[Long context](https://docs.baseten.co/examples/models/capabilities/long-context)

## Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

**Install and sign in to Baseten**

> [tab] macOS or Linux
>
> ```bash Terminal theme={"system"}
> brew tap basetenlabs/baseten
> brew trust --formula basetenlabs/baseten/baseten
> brew install baseten
> ```

> [tab] Windows
>
> Download and extract the Windows archive for the latest Baseten CLI release:
>
> ```powershell Terminal theme={"system"}
> $release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
> $asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
> Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
> Expand-Archive -Force baseten.zip .
> ```
>
> Move `baseten.exe` to a directory on your `PATH`.

For other platforms or a specific version, see the [Baseten CLI install reference](https://docs.baseten.co/reference/cli/baseten/overview.md#install).

```sh theme={"system"}
baseten auth login
```

**Install the OpenAI SDK**

```sh theme={"system"}
uv pip install openai
```

To install without Homebrew, see the [Baseten CLI install reference](https://docs.baseten.co/reference/cli/baseten/overview.md#install).

Pick the model you want to deploy. Each tab is a self-contained recipe.

> [tab] Flash
>
> [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) is a VERIFY-parameter MoE model (VERIFY active per token) with up to VERIFY context.
>
> This preset serves DeepSeek V4 Flash on B200:4 with FP8 KV cache, the `deep_gemm_mega_moe` backend, expert parallelism, and MTP speculative decoding, tuned for low time-to-first-token.
>
> > [card] Hardware
> >
> > B200 × 4
>
> ## Write the config
>
> Create and move into the project directory:
>
> ```sh theme={"system"}
> mkdir deepseek-v4-flash-latency && cd deepseek-v4-flash-latency
> ```
>
> Then create a file named `config.yaml` and paste the following:
>
> ```yaml config.yaml theme={"system"}
> # Registry copy of the config serving the Baseten library listing baseten/deepseek-v4-flash
> # (BIS-LLM). The bis_llm block and environment are the deployed values verbatim; model_name
> # follows the repo convention, resources use instance_type, and the weights entry drops the
> # workspace-specific auth secret because the checkpoint is public and ungated.
> model_name: "model:deepseek-v4-flash preset:bis-latency"
> model_metadata:
>   repo_id: deepseek-ai/DeepSeek-V4-Flash-0731
>   tags:
>     - openai-compatible
>   example_model_input:
>     messages:
>       - role: user
>         content: "What is the meaning of life?"
>     stream: true
>     model: deepseek-ai/DeepSeek-V4-Flash-0731
>     max_tokens: 32768
>     temperature: 1
> environment_variables: {}
> resources:
>   instance_type: B200:4
> bis_llm:
>   version: 0.0.1-20260727180920-c08268b2
>   config:
>     model_name: deepseek-ai/DeepSeek-V4-Flash-0731
>     model_path: /models/deepseek-v4-flash
>     model_path_for_tokenizer: /models/deepseek-v4-flash/tokenizer
>     checkpoint_name: deepseek-ai/DeepSeek-V4-Flash-0731
>     gpuTRTImage: baseten/dynamo-cache-aware-routing:trtllm-rc23-1.3.0rc23-f598f80c4-e3957193bc
>     served_model_name: deepseek-ai/DeepSeek-V4-Flash-0731
>     served_model_name_response: deepseek-ai/DeepSeek-V4-Flash-0731
>     chat_processor: deepseek_v4
>     reasoning_parser: deepseek_v4
>     tool_call_parser: deepseek_v4
>     arguments_as_json: true
>     tensor_parallel_size: 4
>     tokenizer_limit_length: 1048576
>     tokenizer_max_new_tokens_limit: 32768
>     default_thinking_enabled: true
>     default_sampling_params:
>       top_p: 1
>       temperature: 1
>     b10_routing_config:
>       durable_kv_events: false
>       enable_worker_tree: true
>       router_track_residency: true
>       router_overlap_score_weight: 8
>     additional_environment_variables:
>       Worker:
>         TRTLLM_ENABLE_PDL: "1"
>         BAD_TOKEN_ID_SEQ_LEN: "10"
>         BAD_TOKEN_ID_SEQ_CHECK_ENABLED: "1"
>     engine_config:
>       backend: pytorch
>       trust_remote_code: true
>       max_seq_len: 1048576
>       max_input_len: 1048576
>       max_batch_size: 8
>       max_beam_width: 1
>       max_num_tokens: 8192
>       enable_chunked_prefill: true
>       enable_iter_perf_stats: true
>       moe_expert_parallel_size: 4
>       moe_config:
>         backend: TRTLLM
>         use_low_precision_moe_combine: true
>       kv_cache_config:
>         dtype: fp8
>         host_cache_size: 100000000000
>         tokens_per_block: 128
>         enable_block_reuse: true
>         enable_partial_reuse: false
>         event_buffer_max_size: 65536
>         free_gpu_memory_fraction: 0.9
>       speculative_config:
>         decoding_type: DSpark
>         speculative_model_dir: /models/deepseek-v4-flash/worker
>         max_draft_len: 5
>         block_size: 5
>         markov_rank: 256
>         markov_head_type: vanilla
>         mask_token_id: 128799
>         target_layer_ids: [40, 41, 42]
>         allow_advanced_sampling: true
> weights:
>   # Public, ungated checkpoint (MIT); no auth secret. The deployed listing referenced a
>   # workspace-specific secret that does not exist in the registry deploy account.
>   - source: hf://deepseek-ai/DeepSeek-V4-Flash-0731@7872f01b1d1fe23eabc4c98b48bffcef5a386062
>     mount_location: /models/deepseek-v4-flash
> ```
>
> The container loads DeepSeek V4 Flash weights to `/models/deepseek-v4-flash` and serves the OpenAI-compatible API on port 8000. FP8 KV cache and the `deep_gemm_mega_moe` backend keep memory bandwidth in check, and the MTP speculator runs two draft tokens per step to amortize sampling cost.
>
> ## Deploy
>
> Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
>
> ```sh Baseten CLI theme={"system"}
> baseten model push
> ```
>
> ```sh Truss CLI theme={"system"}
> uvx truss push
> ```
>
> You should see output similar to:
>
> ```output theme={"system"}
> Pushing model "deepseek-v4-flash-latency"...
> Uploading model...
> Uploaded model in 0s
> ✨ Model deepseek-v4-flash-latency was successfully pushed ✨
>
>   Model:       deepseek-v4-flash-latency (abc1d2ef)
>   Deployment:  xyz123
>   Environment: production
>
> 🪵 View logs:
>    deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
>    environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
>    app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123
>
> 🚀 Invoke your model:
>    URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
>    CLI:  baseten model predict --model-id abc1d2ef
> ```
>
> `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.
>
> ## Call the model
>
> Your deployment serves an OpenAI-compatible API.
>
> Now call your deployment to run inference:
>
> > [tab] Python
> >
> > ```python main.py theme={"system"}
> > import os
> > from openai import OpenAI
> >
> > client = OpenAI(
> >     api_key=os.environ["BASETEN_API_KEY"],
> >     base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
> > )
> >
> > response = client.chat.completions.create(
> >     model="deepseek-ai/DeepSeek-V4-Flash-0731",
> >     messages=[
> >         {"role": "user", "content": "What is machine learning?"}
> >     ],
> > )
> >
> > print(response.choices[0].message.content)
> > ```
>
> > [tab] cURL
> >
> > ```sh theme={"system"}
> > curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
> >   -H "Content-Type: application/json" \
> >   -H "Authorization: Bearer $BASETEN_API_KEY" \
> >   -d '{
> >     "model": "deepseek-ai/DeepSeek-V4-Flash-0731",
> >     "messages": [
> >       {"role": "user", "content": "What is machine learning?"}
> >     ]
> >   }'
> > ```

> [tab] Pro
>
> [deepseek-ai/DeepSeek-V4-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) is a VERIFY-parameter MoE model (VERIFY active per token) with up to VERIFY context.
>
> This preset serves DeepSeek V4 Pro on B200:8 with FP8 KV cache, the `deep_gemm_mega_moe` backend, expert parallelism, and MTP speculative decoding, tuned for low time-to-first-token at full scale.
>
> > [card] Hardware
> >
> > B200 × 8
>
> ## Write the config
>
> Create and move into the project directory:
>
> ```sh theme={"system"}
> mkdir deepseek-v4-pro-latency && cd deepseek-v4-pro-latency
> ```
>
> Then create a file named `config.yaml` and paste the following:
>
> ```yaml config.yaml theme={"system"}
> # Registry copy of the config serving the Baseten library listing
> # baseten/deepseek-v4-pro (BIS-LLM, TRT-LLM PyTorch backend behind Dynamo cache-aware routing). The bis_llm block,
> # environment, and weights values are the deployed values from BIS Config Registry
> # composer `dsv4-pro-golden` (bcr_version 1.0.9) verbatim; model_name
> # follows the repo convention and resources use instance_type.
> model_name: "model:deepseek-v4-pro preset:bis-latency"
> model_metadata:
>   repo_id: deepseek-ai/DeepSeek-V4-Pro
>   tags:
>     - openai-compatible
>   example_model_input:
>     messages:
>       - role: user
>         content: "What is the meaning of life?"
>     stream: true
>     model: deepseek-ai/DeepSeek-V4-Pro
>     max_tokens: 1024
> bis_llm:
>   config:
>     additional_environment_variables:
>       Worker:
>         BAD_TOKEN_ID_SEQ_CHECK_ENABLED: "1"
>         BAD_TOKEN_ID_SEQ_LEN: "10"
>         TRTLLM_ENABLE_PDL: "1"
>     arguments_as_json: true
>     b10_autoscaling_config:
>       additional_autoscaling_config:
>         metrics:
>           - name: in_flight_tokens
>             target: 130000
>     b10_routing_config:
>       durable_kv_events: false
>       enable_worker_tree: true
>     chat_processor: deepseek_v4
>     checkpoint_name: deepseek-ai/DeepSeek-V4-Pro
>     default_sampling_params:
>       temperature: 1
>       top_p: 1
>     default_thinking_enabled: true
>     engine_config:
>       backend: pytorch
>       enable_chunked_prefill: true
>       enable_iter_perf_stats: true
>       kv_cache_config:
>         dtype: fp8
>         enable_block_reuse: true
>         enable_partial_reuse: false
>         enable_tp_mla_replicated_host_offload: true
>         event_buffer_max_size: 65536
>         free_gpu_memory_fraction: 0.9
>         host_cache_size: 500000000000
>         tokens_per_block: 128
>       max_batch_size: 8
>       max_beam_width: 1
>       max_input_len: 1048576
>       max_num_tokens: 16384
>       max_seq_len: 1048576
>       moe_config:
>         backend: TRTLLM
>         use_low_precision_moe_combine: true
>       moe_expert_parallel_size: 4
>       speculative_config:
>         allow_advanced_sampling: true
>         decoding_type: MTP
>         max_draft_len: 3
>       trust_remote_code: true
>     gpuTRTImage: "baseten/dynamo-cache-aware-routing:trtllm-gpu-f83d539b2a-00b032f66-c62638826d"
>     model_name: deepseek-ai/DeepSeek-V4-Pro
>     model_path: /models/deepseek-v4-pro
>     model_path_for_tokenizer: /models/deepseek-v4-pro/tokenizer
>     reasoning_parser: deepseek_v4
>     served_model_name: deepseek-ai/DeepSeek-V4-Pro
>     served_model_name_response: deepseek-ai/DeepSeek-V4-Pro
>     tensor_parallel_size: 8
>     tokenizer_limit_length: 1048576
>     tokenizer_max_new_tokens_limit: 32768
>     tool_call_parser: deepseek_v4
>   version: 0.0.1-20260601190849-691c46ce
> environment_variables: {}
> # The 865 GB checkpoint and TRT-LLM initialization outlasted the PR deployment
> # startup window; allow the worker to finish before readiness is judged.
> runtime:
>   health_checks:
>     startup_threshold_seconds: 2400
> resources:
>   instance_type: "B200:8"
> weights:
>   # Public, ungated checkpoint; the deployed listing's workspace-specific secret
>   # (hf_access_b10_b10admin) is dropped.
>   - mount_location: /models/deepseek-v4-pro
>     source: "hf://deepseek-ai/DeepSeek-V4-Pro@5607980f3a4b8ea0371b9f11e1848ac41f14979e"
> ```
>
> The container loads DeepSeek V4 Pro weights to `/models/deepseek-v4-pro` and serves the OpenAI-compatible API on port 8000. Tensor parallelism is set to 8 across the B200 fleet, FP8 KV cache and the `deep_gemm_mega_moe` backend hold memory bandwidth, and the MTP speculator runs two draft tokens per step.
>
> ## Deploy
>
> Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
>
> ```sh Baseten CLI theme={"system"}
> baseten model push
> ```
>
> ```sh Truss CLI theme={"system"}
> uvx truss push
> ```
>
> You should see output similar to:
>
> ```output theme={"system"}
> Pushing model "deepseek-v4-pro-latency"...
> Uploading model...
> Uploaded model in 0s
> ✨ Model deepseek-v4-pro-latency was successfully pushed ✨
>
>   Model:       deepseek-v4-pro-latency (abc1d2ef)
>   Deployment:  xyz123
>   Environment: production
>
> 🪵 View logs:
>    deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
>    environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
>    app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123
>
> 🚀 Invoke your model:
>    URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
>    CLI:  baseten model predict --model-id abc1d2ef
> ```
>
> `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.
>
> ## Call the model
>
> Your deployment serves an OpenAI-compatible API.
>
> Now call your deployment to run inference:
>
> > [tab] Python
> >
> > ```python main.py theme={"system"}
> > import os
> > from openai import OpenAI
> >
> > client = OpenAI(
> >     api_key=os.environ["BASETEN_API_KEY"],
> >     base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
> > )
> >
> > response = client.chat.completions.create(
> >     model="deepseek-ai/DeepSeek-V4-Pro",
> >     messages=[
> >         {"role": "user", "content": "What is machine learning?"}
> >     ],
> > )
> >
> > print(response.choices[0].message.content)
> > ```
>
> > [tab] cURL
> >
> > ```sh theme={"system"}
> > curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
> >   -H "Content-Type: application/json" \
> >   -H "Authorization: Bearer $BASETEN_API_KEY" \
> >   -d '{
> >     "model": "deepseek-ai/DeepSeek-V4-Pro",
> >     "messages": [
> >       {"role": "user", "content": "What is machine learning?"}
> >     ]
> >   }'
> > ```

## Next steps

> [card] [Call your model](https://docs.baseten.co/inference/calling-your-model.md)
>
> Endpoint anatomy, authentication, and sync versus async inference

> [card] [Autoscaling](https://docs.baseten.co/deployment/autoscaling/overview.md)
>
> Scale replicas with traffic, including scale to zero