Inference

Troubleshoot common problems during model inference

For HTTP status code and error message meanings (such as what a 502 indicates), see Inference errors.

GPU performance issues

Compare GPU metrics with response time, request volume, concurrent requests, and replica counts over the same period.

High temperature and degraded performance

High temperature with rising latency or falling throughput can indicate thermal throttling. Compare temperature, latency, and throughput with your deployment’s usual readings at similar load on the same GPU type.

• Check recent changes to batching, concurrency, sequence length, and request volume. Use event overlays to check whether configuration changes coincide with the performance regression.
• Temporarily reduce incoming load or per-replica concurrency if latency or throughput degrades. Reducing concurrency can increase queueing, so monitor end-to-end response time.
• If temperature and performance remain abnormal at normal load, contact Baseten Support. Include the deployment ID, GPU type, affected time range, and latency and temperature trends so support can investigate the hardware.

Identify GPU bottlenecks

Compare GPU memory bandwidth utilization and tensor core utilization with latency and throughput. Use the patterns below to choose what to profile before changing your workload.

High memory bandwidth utilization with low tensor core utilization

Memory transfers may limit performance. For example, large language model (LLM) decoding with small batches repeatedly reads model weights and can leave compute capacity unused. Confirm the bottleneck with a GPU profiler before choosing an optimization.

The roofline diagrams plot compute performance against arithmetic intensity: floating-point operations (FLOPs) per byte transferred. Performance is measured in trillions of FLOPs per second (TFLOP/s). The sloped line represents the memory bandwidth limit; the horizontal line represents the compute limit. These diagrams use illustrative values, not measured performance. See NVIDIA’s roofline guide for details.

High tensor core utilization with high latency

Compute may limit performance. For example, processing long prompts during LLM prefill can involve kernels that require substantial computation. Profile expensive operations, then test precision or kernel optimizations supported by your model and runtime.

Near the compute ceiling, more batching alone offers limited throughput improvement.

Low memory bandwidth and tensor core utilization with high latency

Investigate CPU preprocessing, request queueing, network transfers, synchronization, and insufficient batching. GPU metrics alone can’t distinguish these causes.

High memory bandwidth and tensor core utilization

Compare utilization over time with profiler measurements for individual kernels and inference phases. High averages can combine activity from different kernels, phases, or replicas; they don’t establish that both resources limit a single kernel.

Profile each phase before choosing an optimization. After each change, compare throughput and response time under similar traffic. Use your latency and throughput targets to evaluate each change; high utilization alone doesn’t confirm an improvement.

Model I/O issues

Error: JSONDecodeError

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This error means you’re attempting to pass a model input that is not JSON-serializable. For example, you might have left out the double quotes required for a valid string:

baseten model predict --model-name my-model --data 'This is not a string' # Wrong
baseten model predict --model-name my-model --data '"This is a string"'   # Correct

Model version issues

Error: No OracleVersion matches the given query

<Server response: {
    'errors': [{
        'message': 'No OracleVersion matches the given query.',
        'locations': [{'line': 3, 'column': 13}],
        'path': ['model_version']
    }],
    'data': {'model_version': None}
}>

Make sure that the model ID or deployment ID you’re passing is correct and that the associated model has not been deleted.

Additionally, make sure you’re using the correct endpoint:

• Production environment endpoint.
• Development deployment endpoint.
• Deployment endpoint.

Authentication issues

Error: Service provider not found

ValueError: Service provider example-service-provider not found in ~/.trussrc

This error means your ~/.trussrc is incomplete or incorrect. It should be formatted as follows:

[baseten]
remote_provider = baseten
api_key = YOUR.API_KEY
remote_url = https://app.baseten.co

Error: You have to log in to perform the request

<Server response: {
    'errors': [{
        'message': 'You have to log in to perform the request',
        'locations': [{'line': 3, 'column': 13}],
        'path': ['model_version'],
        'extensions': {'code': 'UNAUTHENTICATED_ACCESS'}
    }],
    'data': {'model_version': None}
}>

This error occurs on baseten model predict when the API key is missing or incorrect. To fix it, sign in again with baseten auth login. With the Truss CLI, update your API key in the ~/.trussrc file.

Error: Please check the API key you provided

{
        "error": "please check the api-key you provided"
}

This error occurs when using curl or similar to call the model through its API endpoint when the API key passed in the request header is not valid. Make sure you’re using a valid API key then try again.