Skip to main content
Use the Metrics tab in your model dashboard to compare request load, latency, and resource usage.
  1. Sign in to app.baseten.co, choose Dedicated Inference, and select your model.
  2. Open the Metrics tab.
  3. Use the scope dropdowns to select an environment or deployment, then choose a time range.
This page covers model metrics. For sandboxes, see Inspect sandbox metrics. Environment scope aggregates metrics across every deployment in that environment, which helps you watch a rollout or compare trends across the whole environment. Deployment scope restricts metrics to a single deployment ID for diagnosing one version in isolation. Model dashboard showing inference volume, latency, and resource graphs For exact exported names, types, and labels, see the exported metrics reference. To retrieve metrics with the CLI or API, see View logs and metrics.

​
Customize your view

Use Customize view to choose which graphs appear and how to arrange them:
  1. In the Metrics tab, open Metrics options and choose Customize view.
  2. Select a graph to show or hide it.
  3. Drag graphs in the list to change their order.
  4. Close Customize view to return to the dashboard.
The dashboard saves your layout per model in your browser. Hidden graphs remain in Customize view so you can show them again.

​
Events

Use Events to overlay platform events on your graphs:
  1. In the Metrics tab, open Events.
  2. Turn on Show events and select the event types you want to include.
  3. Hover over an event marker to see its details.
Compare event timestamps with changes in response time or replica count to identify changes worth investigating. Event overlays are available for models. Shared Model API endpoints and training jobs don’t have this overlay. Baseten marks these events:
  • Deployed: a new deployment, with its target environment.
  • Promoted: a deployment promoted to an environment.
  • Promotion control action: a pause, resume, or roll-forward during a promotion.
  • Autoscaling changed: a new replica range or concurrency target.
  • Activated and Deactivated: a deployment turned on or off.
  • Instance type changed: a move to a new instance type.
  • Replica terminated: an individual replica shut down.
  • Environment updated: a change to an environment.

​
Inference volume

Tracks the response rate over time, segmented by HTTP status codes:
  • 2xx: Successful requests.
  • 4xx: Client errors.
  • 5xx: Server errors, including model prediction exceptions.
WebSocket and gRPC deployments report protocol-specific status codes. For WebSocket connections, see WebSocket status codes.

​
Incoming request volume

Tracks the rate of requests arriving at your deployment, before they’re routed to the model. Compare it with Inference volume, which counts responses by status code. Requests can arrive and complete in different time windows, so a temporary difference doesn’t establish that requests failed. To export the same series, see baseten_incoming_inference_requests_total.

​
Response time

Measured at different percentiles (p50, p90, p95, p99):
  • End-to-end response time: Includes cold starts, queuing, and inference. Excludes the network round trip between your client and Baseten.
  • Inference time: Covers model execution, including preprocessing and postprocessing.
  • Time to first byte: Measures the time from request receipt to the first byte Baseten writes back, including any queueing and routing time. It approximates time to first token (TTFT), because Baseten doesn’t parse tokens out of your model’s response. Model API endpoints graph time to first token instead.
To see how long each replica took to start and which phase dominated, use the Startup time card described in Startup times.

​
Request and response size

Measured at different percentiles (p50, p90, p95, p99):
  • Request size: Request payload size in bytes.
  • Response size: Response payload size in bytes.
Payload size also includes formatting and non-text content. Use engine token metrics when you need token counts.

​
Replicas

Tracks how many replicas the autoscaler is targeting and how many you have:
  • Desired: The number of replicas your autoscaling settings call for at the current concurrent request volume. BIS-LLM deployments target in-flight tokens instead.
  • Starting: Waiting for resources or loading the model.
  • Active: Ready to serve requests. For development deployments, a replica counts as active while it runs the live reload server.
Compare Desired with Active and Starting to see whether available capacity is catching up with the autoscaler’s target. A persistent gap needs investigation into deployment status, startup logs, and capacity; the graph alone doesn’t identify the cause. For scaling controls, see Autoscaling. For a persistent gap or slow scale-up, see Autoscaling troubleshooting. The exported metrics reference defines the replica count series. To distinguish replicas by their Kubernetes Ready condition, see baseten_pod_readiness.

​
Restarts

Tracks the cumulative number of times the model container has restarted. Application crashes, out-of-memory kills, or failed liveness probes can cause restarts. Frequent restarts usually indicate one of:
  • A crash in load() or in your model code.
  • An out-of-memory event: check the Memory usage graph.
  • A liveness probe failing under load: review restart_threshold_seconds and any custom health check logic.

​
Concurrent requests

Total concurrent inference requests across replicas, including both requests currently being serviced and requests waiting to be processed. Async inference requests aren’t included in this metric. Concurrency drives request-based autoscaling. BIS-LLM deployments use in-flight tokens. For the full metric definition and labels, see baseten_concurrent_requests. This metric is a sampled gauge. Short requests can begin and finish between samples, while inference volume counts completed responses over a time window. For steady traffic, the relationship is approximately: average concurrency ≈ requests per second × average end-to-end latency For example, 600 requests per minute at 80 ms average latency implies about 0.8 concurrent requests. Individual samples can still read zero. Compare concurrency with request rate and latency over the same period before interpreting a zero as no traffic.

​
CPU usage and memory

CPU usage shows average CPU usage across replicas, in cores. CPU memory shows average working-set memory usage across replicas. Aggregation can hide brief spikes or one replica using more resources than the others. Compare resource usage with latency and deployment logs to assess whether a resource limit affects requests before changing instance size.

​
GPU usage and memory

  • GPU usage: Average GPU utilization across replicas, as a percentage.
  • GPU memory: Average GPU memory used across replicas.
  • GPU temperature: Average GPU temperature across replicas, in degrees Celsius.
  • GPU tensor core utilization: Average tensor core utilization across replicas, as a percentage. Availability depends on GPU profiling support.
  • GPU memory bandwidth utilization: Average memory read and write activity across GPUs, as a percentage.
Compare these graphs with response time, request volume, concurrent requests, and replica counts over the same period. Use Customize your view to show the graphs you need. GPU usage and tensor core utilization measure different activity, so their values don’t need to match. See baseten_gpu_utilization and baseten_gpu_memory_used for export units and hardware-specific behavior. The exported metrics reference lists which dashboard metrics you can collect externally. For diagnostic checks and corrective actions, see GPU performance issues.

​
Organization GPU usage

The GPU graphs above follow the selected model scope. To see how many GPUs your whole workspace is using at once, organization admins can open the GPU usage tab in Organization settings. This view counts active GPUs (ready replicas multiplied by the GPUs each replica uses), aggregated across every model and deployment in the organization. To view organization GPU usage:
  1. Sign in to your workspace at app.baseten.co and open Organization settings.
  2. Choose the GPU usage tab.
  3. Use Group by to choose GPU type (the default) or Model.
  4. Use the GPU filter to select the GPU types you want to view.
  5. Use the time-range selector to choose a window up to the last 7 days.
  6. To clear the GPU filter, choose Reset.

​
vLLM and SGLang metrics

When your deployment serves an LLM with vLLM or SGLang, Baseten surfaces engine-native metrics in the Metrics tab alongside the standard ones. These graphs report what the inference engine itself measures: token throughput, decode latency, cache usage, and the number of requests running or queued. See LLM token speed for how decode speed differs from aggregate token throughput.

​
How detection works

Baseten scrapes your container’s /metrics endpoint and looks for metrics that match the vLLM or SGLang format. When it finds them, the matching graphs appear in the Metrics tab automatically. Your engine must expose its metrics first. For SGLang, start the server with --enable-metrics; see SGLang’s metrics configuration. If engine graphs are absent from Customize view, confirm that the selected deployment exposes Prometheus metrics on /metrics. For custom servers, this route passes through to your server. Detection uses cached results. Newly emitted metrics don’t make graphs appear immediately. If graphs remain unavailable, check the endpoint and selected deployment, then contact support.

​
Show and hide graphs

Many of these engine graphs are hidden by default. Turn them on with Customize your view. The available data depends on what your engine version emits. Latency graphs show p50, p90, p95, and p99. Counter graphs show changes over each chart interval, or a rate where the graph specifies tokens per second.

​
Export engine metrics

The Metrics tab shows a selection of engine graphs. You can also collect supported engine metrics through the metrics export endpoint. See vLLM and SGLang metrics for the exact exported names, types, labels, and availability conditions.

​
BIS-LLM metrics

When your deployment runs on the BIS-LLM engine, the Metrics tab adds engine graphs alongside the standard metrics above. The available graphs depend on the deployment configuration and enabled features.
  • Token throughput: input and output tokens over time, with per-request distributions for input tokens, output tokens, and output tokens per second.
  • Decode tokens per second: how quickly individual streams generate output after the first token.
  • KV cache hit rate: the cache hit rates your workers observe. Compare hit rate with time to first byte to investigate changes in prefix reuse and prefill work.
  • Time to first byte: on BIS-LLM deployments this graph is measured inside the engine, covering engine queueing and prefill but not platform routing, unlike the standard time to first byte, which includes routing time. Response time still covers the end-to-end view.
  • In-flight tokens per worker: the load signal the BIS-LLM autoscaler acts on, covering tokens being processed and queued at the router.
An acceptance rate graph is available when the deployment exposes speculative decoding metrics. Hide or show individual graphs with Customize your view.

​
Export engine metrics

Supported BIS-LLM export series use the baseten_llm_* prefix. Collect them in your observability stack through the metrics export endpoint. The reference includes accepted and draft token counters for speculative decoding. For the full list with types and labels, see BIS-LLM metrics.

​
LLM token speed

Distinguish the speed of an individual response from the total number of tokens a deployment generates:
  • Decode tokens per second: Output generation speed after the first token.
  • Output tokens per second per request: Output tokens divided by the request’s full processing time, including first-token delay.
  • Output token throughput: Total output tokens generated per second across requests. Some engine graphs call this generation throughput.
Use decode speed to investigate streaming performance. Use output tokens per second per request to include first-token delay. Use aggregate output token throughput to measure total serving capacity. The BIS-LLM and Model APIs time-per-output-token histogram records chunks that contain new output tokens after the first chunk. Each observation measures time since the previous chunk divided by new output tokens. A chunk containing several tokens has the same weight as a single-token chunk. Longer responses generally contribute more observations. The decode mean derived from this histogram is 1 / mean(time per output token). It isn’t the arithmetic mean of the tokens-per-second values for each chunk. vLLM and SGLang define their own engine histograms; see their links in the engine metrics reference.

​
Compute decode tokens per second from exports

Baseten exports the baseten_llm_time_per_output_token_seconds histogram with these series suffixes: _bucket_total, _count_total, and _sum_total. Because throughput inverts latency, high latency quantiles describe the slow end of throughput. For example, the reciprocal of p99 latency estimates p1 throughput. Histogram quantiles are estimates based on bucket boundaries. These queries aggregate replicas within one deployment over 5 minutes. Replace <DEPLOYMENT_ID> with your deployment ID. The histogram must be available for your organization and have observations in that window. Mean decode rate derived from the histogram:
Median (p50) decode tokens per second:
Estimated slow tail (p1 throughput from p99 latency):

​
Model APIs metrics

Model API endpoints have their own Metrics tab. Because you call a shared endpoint rather than run your own replicas, the tab graphs request and token metrics, not the replica and hardware metrics above.
  • Inference volume: response rate over time, segmented by HTTP status code.
  • Rate limited requests: rate of requests the endpoint rejects for exceeding your rate limits.
  • End-to-end response time: time from request receipt to the last byte of the response, including queueing and inference.
  • Time to first token: first-token latency for the endpoint. Its measurement depends on the serving path; compare it with end-to-end response time when investigating delays.
  • Request size and Response size: request and response payload size distributions.
  • Decode tokens per second: per-stream output token generation rate after the first token. See LLM token speed.
  • Total tokens: tokens per second the endpoint processes for your workspace.
The latency and size graphs report the p50, p90, p95, and p99 percentiles.

​
Export Model API metrics

Supported Model API export series use the baseten_model_api_* prefix. Collect them in your observability stack through the metrics export endpoint. That includes per-status-code request counts, so you can alert on 429 responses, and the split of cached versus uncached input tokens. For the full list with types and labels, see Model API metrics.

​
Async queue metrics

  • Time in async queue: Time spent waiting in the queue before processing begins, at p50, p90, p95, and p99.
  • Async queue size: Requests waiting for a replica to pick them up.
  • Webhook requests: Rate of async webhook delivery requests, grouped by HTTP status code.
  • Webhook latency: Duration of webhook delivery calls, including retries within each call, at p50, p90, p95, and p99. Failed calls also contribute observations.
Queue size is a sampled gauge. Requests that pass through the queue between samples might not appear in it. Sustained queue growth means arrivals are outpacing processing; compare queue time, replica counts, and logs before tuning async concurrency.

​
Use metrics for autoscaling

Compare request load, response time, and replica counts over the same period. Use event overlays and logs to investigate changes. See Autoscaling for scaling controls, Autoscaling engines for engine-specific load signals, and Autoscaling troubleshooting for diagnosis.