Train a coding agent on Terminal-Bench

Train GLM-5.3-Flash with asynchronous reinforcement learning as a multi-turn bash agent on Terminal-Bench 2.1 tasks in Blaxel sandboxes.

This example trains GLM-5.3-Flash with reinforcement learning (RL) as a coding agent. The agent works through Terminal-Bench 2.1 tasks with one bash tool, each task in its own Blaxel sandbox. Each task’s own tests grade the result.

The tasks are Harbor tasks, so Harbor builds each environment and runs each task’s verifier. Start with the tool-using agent example if you haven’t run RL with the Loops cookbook before. This run is much larger, and some tasks take hours.

​

Prerequisites

• A Baseten workspace with Loops access to zai-org/GLM-5.3-Flash, and its API key. GLM-5.3-Flash trains in either availability mode.
• A Blaxel workspace and API key, with capacity for about 50 concurrent sandboxes at 8 GB of memory each.
• uv and Python 3.12 or later.

​

Install the cookbook

Install the Loops cookbook from PyPI into a uv project. The cookbook needs Python 3.12 or later, installs the baseten-loops SDK, and provides the loops command. loops runs both recipes; each training config names its recipe with recipe = "rl" or recipe = "sft". Create a project first if you don’t have one:

uv init --python 3.12 loops-project
cd loops-project
uv add baseten-loops-cookbook

The package includes the cookbook’s examples. Run one using its bundled config path, such as examples/counting/train.toml, from your project’s directory. To change an example, copy its directory into your project at the same path. A config in your project wins over the installed one, and loops check prints the directory of the config it used.

Create a .env file in your project’s directory and set BASETEN_API_KEY to a workspace API key. The commands load .env from the training config’s directory or the directory you run them from. Never commit .env.

A run uses the serverless availability mode when serverless supports the model and the run’s LoRA rank and sequence length, and the dedicated mode otherwise. To choose a mode, add BASETEN_LOOPS_SCOPE=serverless or BASETEN_LOOPS_SCOPE=deployment to .env.

Install Harbor with its Blaxel environment, and add BL_WORKSPACE and BL_API_KEY to .env:

uv add "baseten-loops-cookbook[harbor-blaxel]"

The config mirrors metrics to Weights & Biases. To use it, install the extra with the following command, then add WANDB_API_KEY to .env:

uv add "baseten-loops-cookbook[harbor-blaxel,wandb]"

To turn it off, start the run with the following command:

uv run loops run examples/terminal_bench/train.toml --output.wandb_project ""

Every setting in the config has a command-line flag named by its section and key, so you can change the example without copying it. uv run loops run --help lists the flags.

​

Read the agent

examples/terminal_bench/terminal_agent.py defines TerminalAgent, a Harbor agent that calls the policy directly. Its system prompt describes a software engineer in a sandbox. Its one tool runs a bash command in the task’s environment and returns the exit code, stdout, and stderr, each truncated to max_output_bytes. The agent loops until the model replies without a tool call or reaches max_turns.

TerminalAgent discards a reply the token limit cut and samples the turn again, up to three times, so a cut generation never trains. After three cut attempts, the episode ends as truncated and is still graded. The tool-using agent makes the opposite choice: it ends and grades the episode, and the cut generation trains with the rest.

examples/terminal_bench/harbor.yaml pins the Terminal-Bench 2.1 dataset to one commit, selects the cookbook’s Blaxel environment, and passes the agent’s limits:

examples/terminal_bench/harbor.yaml

datasets:
  - repo: https://github.com/harbor-framework/terminal-bench-2-1@7131e4375048a0e408a8fb404b5f499d726b695b
    path: tasks
agents:
  - import_path: terminal_agent:TerminalAgent
    kwargs:
      max_turns: 64
      command_timeout: 120
      max_output_bytes: 16384

The job must use loops_cookbook.harbor.blaxel:BlaxelEnvironment rather than Harbor’s own Blaxel environment. The cookbook’s class records every sandbox in the run’s resource journal, so loops cleanup can release them after a crash.

​

Read the training config

examples/terminal_bench/train.toml trains asynchronously: the sampler collects rollouts for the next update while the trainer applies the current one. The settings that shape the run:

examples/terminal_bench/train.toml

[model]
base_model = "zai-org/GLM-5.3-Flash"
renderer = "loops_cookbook.llm.renderer_extensions.glm53:renderer"   # required for this model
max_seq_len = 262144
lora_rank = 32
renderer_chat_template_kwargs = { reasoning_effort = "low", clear_thinking = false }

[training]
mode = "async"
max_steps = 50
learning_rate = 0.00001
adam_epsilon = 0.00000001
router_replay = false
group_size = 8
max_group_losses = 2
groups_per_step = 4
max_consecutive_unusable_groups = 100
max_steps_off_policy = 3
max_policy_delta_in_trajectory = 3

[compute]
spot_job = true
sampler_replica_affinity = true

• Each update trains 4 groups of 8 attempts, for 50 updates.
• max_steps_off_policy = 3 lets a trained rollout come from a policy up to three updates behind the trainer. Async runs trade that staleness for throughput. Read freshness/lag_mean to see the lag the run actually trained on.
• max_group_losses = 2 lets a group train when up to 2 of its 8 sandboxes fail.
• spot_job and sampler_replica_affinity are account-specific opt-ins. Spot capacity can be reclaimed mid-run, which stops the run and nothing restarts it. Sticky sessions need sampler affinity routing enabled for your Loops workspace; sampling requests fail without it.

To run without them, start the run with the following command:

uv run loops run examples/terminal_bench/train.toml \
  --compute.spot_job false \
  --compute.sampler_replica_affinity false

spot_job requests spot capacity for deployments; it doesn’t choose between the serverless and dedicated modes.

​

Scale out the run

GLM-5.3-Flash uses the serverless mode by default when your workspace has access, and needs no capacity request. A long asynchronous run that needs more training and rollout throughput belongs in the dedicated mode, where you set the trainer and sampler replica counts.

To use the dedicated mode, add the following line to .env:

.env

BASETEN_LOOPS_SCOPE=deployment

Then raise the replica counts when you start the run, for example as follows. trainer_replicas adds data-parallel trainer replicas for update throughput, and sampler_replicas adds sampler replicas for rollout throughput. Each trainer replica is a full copy of the model’s GPU allocation, and the deployment bills for GPU time, including idle time:

uv run loops run examples/terminal_bench/train.toml \
  --compute.trainer_replicas 2 \
  --compute.sampler_replicas 4

To plan capacity for a large run in the dedicated mode, request more access. Scale the trainer explains how batches split across replicas.

​

Run the example

To check the configuration, run the following command. It downloads the dataset, validates the job and agent options, and launches nothing. It warns about any task that declares more disk than the sandbox provides:

uv run loops check examples/terminal_bench/train.toml

To train, run the following command. With serverless Loops, Baseten runs the trainer and sampler and bills for the tokens the run trains and samples, as long as serverless supports the model, the LoRA (low-rank adaptation) rank, and the sequence length. Otherwise, the run uses dedicated capacity, which bills for GPU time. The run also starts one Blaxel sandbox per running attempt: 48 at a time, plus up to 5 started ahead so a freed slot is filled at once. Sandboxes bill while they run:

uv run loops run examples/terminal_bench/train.toml

Tasks vary from minutes to hours. The config allows each rollout up to 6 hours (rollout_timeout_seconds = 21600) and reports pipeline progress every 60 seconds, so a slow phase is distinguishable from a stall. The run writes run.json and steps.jsonl to its run directory.

​

Read the results

Monitor reward/mean, discard/groups/no_variance, and rollout/* in W&B or in the run directory’s steps.jsonl. Terminal-Bench tasks vary widely in difficulty, so expect many no_variance groups from tasks the starting model always fails. For help reading the metrics, see Check whether an RL run is learning.

To tell whether the trained agent is a better coding agent, score a saved checkpoint on tasks it didn’t train on; training reward measures only the tasks the run trained on. For how to run that evaluation, see Evaluate a model.

​

Add a loss on command output

examples/terminal_bench/train_echo.toml runs the same job with ECHO, a custom loss defined in echo_loss.py. It adds a next-token loss on the output the agent’s commands print, weighted by observation_weight, to the importance-sampling loss on the agent’s own tokens. The module’s docstring describes where it differs from the paper.

To install the loss’s extra, run the following command:

uv add "baseten-loops-cookbook[harbor-blaxel,custom-loss]"

To check the configuration, run the following command:

uv run loops check examples/terminal_bench/train_echo.toml

To start the run:

uv run loops run examples/terminal_bench/train_echo.toml

​

Resume a stopped run

If the run stops before its last update, continue it with --resume. The command finds the run of this config that trained most recently, and continues from the last checkpoint it saved:

uv run loops run examples/terminal_bench/train.toml --resume

max_steps counts the updates of the whole run, so the resumed run makes only the updates that are left. To check the resumed run first, run uv run loops check examples/terminal_bench/train.toml --resume. If no run has trained, or the run saved no checkpoint, the command exits with code 2 and starts nothing.

​

Shut down the run

loops run leaves the Loops run active. On serverless, deactivation is optional because idle runs incur no token charges. On dedicated capacity, deactivate the run when you finish to stop GPU billing without waiting for automatic shutdown. To end the run and its paired sampler in either mode, run the following command with your run directory:

uv run loops deactivate "runs/terminal-bench/<run directory>"

The command prints the deactivated run ID. Saved checkpoints remain available.

​

Release sandboxes after a crash

Sandboxes bill until they’re deleted. If the run crashed, release its sandboxes with the following command:

uv run loops cleanup examples/terminal_bench/train.toml \
  --state-dir "runs/terminal-bench/<run directory>/state"

​

Deploy the checkpoint

The run’s sampler checkpoint is ready to serve. The deployment loads the adapter on the base weights, downloads the base weights with the hf_access_token in workspace secrets, and bills for its GPU while it’s live. With the Baseten CLI installed, list the run’s checkpoints and deploy the one you want:

baseten loops checkpoint list --run-id "<run_id>"
baseten loops checkpoint deploy --checkpoint-id "<checkpoint_id>"

The deploy command prompts for a model name, GPU type, GPU count, and the Hugging Face secret name. Deploy a checkpoint covers the REST API and dashboard paths.

​

Next steps

• Design RL environments and rewards: Graders, task sizing, and exploits.
• Check whether an RL run is learning: Metrics for long, asynchronous runs.
• Deploy a checkpoint: Serve the trained adapter.