Audio Input Preprocessing¶

This page describes audio supplied to multimodal chat and generation through --audio or an API content part. It is separate from the Whisper/Kokoro OpenAI Audio API.

Shared Input Contract¶

CLI files and server-resolved bytes pass through the same strict RIFF/WAVE decoder. It accepts PCM16 or float32 WAV, rejects empty, malformed, non-finite, or over-budget input, averages channels to mono, and clamps samples to finite f32 values in [-1, 1]. Clip order and one-based placeholder numbering are preserved.

The checkpoint policy is derived from config.json and processor_config.json or preprocessor_config.json. It supplies the target sample rate, duration/sample limits, feature frame shape, channel/dtype requirements, clip count, aggregate encoded-byte bounds, and host-memory budgets. Invalid policy metadata fails model startup; an invalid request fails instead of silently becoming text-only.

Qualified Families¶

Family	Processing policy	Default request bounds
Phi-4 Multimodal	Mono float32, published SpeechLib integer-ratio resampler, numbered placeholders	Up to 16 clips, 30 minutes and 28,800,000 normalized samples in aggregate, 256 MiB encoded input
Gemma 3n	Mono float32 at 16 kHz, deterministic linear resampling, 188 soft tokens per clip	Up to 16 clips, 30 seconds and 480,000 normalized samples in aggregate, 64 MiB encoded input

The Phi policy intentionally preserves its reference processor's integer-ratio behavior rather than replacing it with the generic linear resampler. Other audio-capable families may own a model-specific front end while still using the shared acquisition and admission boundary.

Server Acquisition and Limits¶

Before model-specific processing, server acquisition enforces an absolute ceiling of 16 clips and 500 MiB of encoded audio per clip and per request. URL Content-Length and base64-size preflights reject obviously oversized input before large allocation. The loaded model's stricter encoded bytes, duration, samples, frames, clips, retained-result, working-memory, and prepared-result budgets are then applied as request aggregates.

Image/audio URL and file access is not sandboxed

The server fetches http:// and https:// image and audio URLs and reads file:// or bare local paths on the client's behalf, so an API key authenticates the caller without confining the server's network or filesystem access. Payload and model budgets are enforced, and two destination policies apply: a remote URL resolving to a non-public address is refused unless MLXCEL_ALLOW_PRIVATE_MEDIA_URLS=1, and a local file is readable only under a configured --media-path root. There is still no hostname allowlist or HTTPS-only switch, and --allowed-media-domains does not exist, so expose multimodal endpoints only to trusted clients or enforce destination policy with a reverse proxy, egress firewall, and container or OS sandbox. Local video paths have a separate fail-closed MLXCEL_VIDEO_DIR_ALLOWLIST. See Media inputs.

Cancellation and Memory Accounting¶

Cancellation is checked while decoding, resampling, and extracting CPU features. Encoded input and host working memory are reserved through an RAII guard from admission until the result is consumed, cancelled, or dropped, which prevents failed requests from leaking the shared budget. After a request enters model execution, cancellation remains cooperative and may not stop a device operation already in flight.

The server exports audio admission, byte, duration, frame, cancellation, and stage-latency counters when metrics are enabled. Use them to distinguish acquisition rejection, preprocessing pressure, and model execution latency.

CLI Examples¶

# Phi-4 Multimodal audio question
mlxcel generate -m ./models/phi-4-multimodal \
  --audio ./meeting.wav \
  -p "Summarize the discussion." -n 200

# Gemma 3n audio input
mlxcel run ./models/gemma-3n-e4b-it \
  --audio ./clip.wav \
  -p "What do you hear?"

Use a standards-compliant PCM16 or float32 WAV. For compressed input, convert it before invoking mlxcel, for example with ffmpeg -i input.mp3 -ac 1 output.wav.