Loops

Resolve capacity, quota, and checkpoint errors and stop interrupted Loops runs.

Resolve capacity, quota, and checkpoint errors and stop a run after an interruption.

​

Resolve capacity and quota errors

In the serverless mode, runs share a capacity pool that Baseten manages for you, so a new run starts without reserving GPUs. A capacity 429 means the request couldn’t run at that moment. It succeeds once capacity frees up, your limit rises, or the run gets GPUs of its own, and the response tells you which applies:

Response	What refused it	What to do
429, Loops is at capacity	The capacity pool Baseten manages is fully committed.	Retry until capacity frees up, or run it in the dedicated mode, which gives the run GPUs of its own.
429, details.limit of org_cap	Your organization reached its limit of concurrent runs and samplers. The pool may still have room.	Retrying won’t help. Deactivate a run you no longer need, or request a higher limit.
429, Could not schedule trainer. Please try again later.	A dedicated run found no eligible GPUs for the GPU type and count it asked for.	Retry later, or ask for a different GPU type or count.
403, Your organization has reached its Loops trainer capacity.	Your organization’s dedicated trainer quota is full.	Deactivate a dedicated run, or request a higher limit.

The SDK retries a capacity refusal for you when it carries a Retry-After header and a details.reason of loops_capacity. It backs off with jitter, waiting never less than Retry-After and never more than a minute between attempts, for up to ten attempts, before raising LoopsCapacityError. The error’s response holds the original details. Loops never refuses a run or sampler it has already started for capacity, so a capacity 429 comes from creating a run or standalone sampler, or from one that sat idle long enough for another to take its place.

​

Retry until capacity frees up

Capacity frees up as other runs finish, and the SDK’s retries cover the first few minutes of that. To wait longer, catch LoopsCapacityError and back off on your own schedule:

from baseten.loops import LoopsCapacityError, ServiceClient

try:
    training_client = ServiceClient().create_lora_training_client(
        base_model="Qwen/Qwen3.6-35B-A3B",
        scope="serverless",
    )
except LoopsCapacityError as error:
    details = error.response.json().get("details", {})
    if details.get("limit") == "org_cap":
        raise  # Your own concurrency limit: free a run or raise the limit.
    # Capacity was full: wait and try again later.

Retrying an org_cap refusal won’t help. It clears when a run or sampler ends or your limit rises.

​

Request more capacity

Baseten can raise the limits on your organization, and a limit change takes longer than the SDK’s retry window. Ask for what a run needs before creating it, and send these details through the Loops request form:

• The models and sequence lengths you train.
• How many runs and samplers you need at the same time.
• When you need the capacity, and for how long.

If a launch or a training campaign needs more than the pool allows, work with us while you plan it, through the request form or your Baseten representative, rather than after a run is refused.

​

Move the run to the dedicated mode

Set scope to deployment to give the run its own trainer and sampler deployments, billed for GPU time, so it no longer shares that capacity:

training_client = ServiceClient().create_lora_training_client(
    base_model="Qwen/Qwen3.6-35B-A3B",
    scope="deployment",
)

The dedicated mode has capacity limits of its own: 429 with Could not schedule trainer means no eligible GPUs are free for the requested shape, and 403 means your organization’s trainer quota is full. See Dedicated Loops.

​

Run has no deployment ID or deployment logs

A serverless run has a null deployment ID, and its sampler has null model and deployment IDs. This is expected. Check the run’s status with baseten loops run describe.

baseten loops run logs works only for dedicated runs and returns an error for serverless runs and samplers. Retain the run ID, operation, HTTP status, and full SDK error when contacting Baseten support. baseten loops usage reports GPU allocations, not serverless consumption or spend.

​

Handle SDK request errors

The SDK treats provisioning errors and requests to a running trainer or sampler differently. Use the operation and error message to decide whether to wait or inspect the run:

Request	Status or error	SDK behavior
Creating a run or standalone sampler	429 with Retry-After and details.reason of loops_capacity	Retries with jittered backoff, never sooner than Retry-After, for up to ten attempts, then raises LoopsCapacityError.
Trainer requests or sampler deployment requests using readiness retries	400, 404, 408, or 502	Retries with backoff until ready_timeout expires, except permanent rejections.
Sampling in the dedicated mode	503	Retries with backoff until ready_timeout expires. A retry can restart generation.
Serverless sampling submission	408, 502, or 504	Retries with the same idempotency key until ready_timeout expires.
Serverless sampling submission or result polling	503 with Retry-After	Waits for the requested delay before retrying.
Trainer operation submission or retrieval	ServerShutdownError (503)	Reports that the server is shutting down instead of treating it as a readiness retry.

Gateway rejections, invalid requests, and deactivated models fail without readiness retries. The SDK retries only the capacity refusal described in capacity and quota errors; every other 429, such as the gateway’s usage limit, is raised as soon as it arrives.

Read sampling retry warnings to identify repeated generation attempts. For an interrupted trainer operation, inspect the run’s status before restarting work. On a serverless sampling timeout or cancellation, the SDK attempts to cancel the sample if it’s still queued.

​

Load a checkpoint trained with a different LoRA configuration

TrainingClient.create_sampling_client(model_path) raises httpx.HTTPStatusError with a 400 response when the checkpoint and the current run use different LoRA configurations:

Output

Checkpoint was trained on LoRA config <CHECKPOINT_VERSION>, but this run uses <RUN_VERSION>. Create a sampling client for the checkpoint from the service client instead.

A run’s paired sampler uses the run’s LoRA configuration, so it can’t load a checkpoint trained with another one. To sample from the checkpoint, create a standalone sampler with ServiceClient.create_sampling_client():

sampling_client = service_client.create_sampling_client(model_path="bt://...")

​

Resolve accelerator errors

If provisioning a deployment fails with either of these messages, contact your organization admin or Baseten support:

• accelerator priority ... has no overlap with your organization's allowlist: The requested GPU type isn’t permitted by your organization’s configuration.
• has no candidate on those accelerators: The permitted GPUs don’t support the requested model and sequence length.

Include the model ID, maximum sequence length, and full error message.

​

Stop a run after an interruption

An SDK timeout doesn’t deactivate the run. If you no longer need a dedicated run, deactivate it to stop GPU billing without waiting for automatic shutdown. An idle serverless run incurs no token charges; deactivate it when you want to end the run and its paired sampler.

Use the run ID saved by your script. If you only have a session ID, see Find a run from its session ID.

​

Find a run from its session ID

The training client can time out before returning a run ID. If your script saved a session_id, list your runs and find the matching session.

Baseten CLI

To find a run from its session ID:

List your runs and filter by the saved session ID. Replace <session_id> with that value. --all includes inactive runs:

baseten loops run list --all \
  --jq '.runs[] | select(.session_id == "<session_id>") | {id, status}'

REST API

To find a run from its session ID:

Call List runs:

curl --fail-with-body \
  "https://api.baseten.co/v1/loops/runs" \
  --header "Authorization: Bearer $BASETEN_API_KEY"

In the response’s runs array, find the entry whose session_id matches your saved value.

Use the matching run’s id to inspect the run or list its checkpoints.