Audio

Send audio recordings alongside text prompts to audio-capable models on Model APIs

Model APIs support audio input on audio-capable models. These models accept audio alongside text in the same request, using the audio_url content type, and process both modalities together: they can transcribe speech, answer questions about audio content, or reason over a recording and a text prompt at once.

To deploy dedicated models for batch or real-time transcription, or for speech generation, see Speech and audio.

Not all models support audio. Check the table below before sending audio inputs.

Supported models

Model	Slug
Muse Spark Open	meta-models/muse-spark-open

Send an audio request

Use the audio_url content type to include audio in your messages. The request shape matches vision requests with audio_url in place of image_url.

Each model has its own audio requirements:

Model	Requirements
Muse Spark Open	WAV, MP3, FLAC, OGG (Vorbis or Opus), or M4A/AAC; 8–48 kHz; mono, stereo, and multichannel audio are downmixed; up to 4 files, 25 MB and 5 minutes each; 100 MiB request-body limit

The combined size of images, videos, and audio must fit within the 200 MB total-media limit. Large combinations can also exceed the encoded-media budget.

Baseten retrieves audio URLs from the inference service. Use an HTTP or HTTPS URL reachable from Baseten’s environment, such as your own object storage. Muse Spark Open follows redirects and rejects localhost, internal, and cloud-metadata IP addresses. Prefer stable, direct HTTPS links. Base64 data: URLs also work, but they count toward the request-body limit, so pass audio by URL when you can.

Send a recording alongside a text prompt like this:

Python

audio.py

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://inference.baseten.co/v1",
    api_key=os.environ["BASETEN_API_KEY"],
)

response = client.chat.completions.create(
    model="meta-models/muse-spark-open",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Summarize what is said in this recording.",
                },
                {
                    "type": "audio_url",
                    "audio_url": {
                        "url": "https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav",
                    },
                },
            ],
        }
    ],
)

print(response.choices[0].message.content)

JavaScript

audio.js

import OpenAI from "openai";

const client = new OpenAI({
    baseURL: "https://inference.baseten.co/v1",
    apiKey: process.env.BASETEN_API_KEY,
});

const response = await client.chat.completions.create({
    model: "meta-models/muse-spark-open",
    messages: [
        {
            role: "user",
            content: [
                {
                    type: "text",
                    text: "Summarize what is said in this recording.",
                },
                {
                    type: "audio_url",
                    audio_url: {
                        url: "https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav",
                    },
                },
            ],
        },
    ],
});

console.log(response.choices[0].message.content);

cURL

Request

curl https://inference.baseten.co/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "meta-models/muse-spark-open",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Summarize what is said in this recording."
          },
          {
            "type": "audio_url",
            "audio_url": {
              "url": "https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav"
            }
          }
        ]
      }
    ]
  }'

Pricing

There is no additional per-request fee for audio. Audio is converted to input tokens and priced at the model’s standard input rate. Longer recordings produce more tokens and cost more to process. See the Model APIs pricing page for per-model rates.

Next steps

Vision

Send images and videos alongside text

Chat Completions reference

Full request and response schema for the audio_url content type