OpenAI Audio API¶

mlxcel implements the OpenAI-compatible speech, transcription, and translation routes. Audio checkpoints occupy a dedicated audio-only server process: use Whisper for speech-to-text (STT) or Kokoro for text-to-speech (TTS), and run a separate process for chat or text generation.

Endpoints¶

Each route also has an alias with the /v1 prefix removed, such as /audio/speech.

If the loaded checkpoint does not provide the requested capability, the route returns HTTP 501 after validating the request.

Nemotron VoiceChat, a full-duplex speech-to-speech model, does not use these routes. It is served over a WebSocket; see Realtime VoiceChat API.

Starting an Audio Server¶

Whisper STT¶

The checkpoint must declare model_type: "whisper" and contain config.json, tokenizer.json, and SafeTensors weights. Native MLX and Hugging Face WhisperForConditionalGeneration key layouts are accepted.

mlxcel-server -m ./models/whisper-base

Input is currently WAV only. mlxcel converts PCM16 or float32 audio to mono, resamples it to 16 kHz, and splits audio longer than 30 seconds into consecutive windows. Whisper currently supports non-quantized fp16/f32 checkpoints and greedy decoding; beam search, timestamps, VAD chunking, MP3, and FLAC are not available.

Kokoro TTS¶

Kokoro is detected from the istftnet configuration block or kokoro-v1_0.safetensors. The directory must contain config.json, the Kokoro weights, and voices/*.safetensors voice packs. A Kokoro checkpoint needs no tokenizer file, because the phoneme vocabulary comes from config.json; /tokenize, /detokenize, and /apply-template answer 501 on such a server, like the generation routes. If the checkpoint cannot be loaded, for example because vocab or a weight file is missing, the server exits at startup instead of running without a TTS slot.

mlxcel-server -m ./models/kokoro-82m

The default voice is af_heart. A safe voice name selects voices/<name>.safetensors; unknown or unsafe names fall back to the default. The built-in grapheme-to-phoneme front end is American English. Input is truncated at 4,096 characters and the acoustic model consumes at most 510 phoneme tokens. WAV is the only output format.

Text-to-Speech¶

curl -s http://localhost:8080/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"kokoro","input":"Hello from mlxcel.","voice":"af_heart","speed":1.0,"response_format":"wav"}' \
  --output speech.wav

A successful response is audio/wav with a PCM16 RIFF body.

Transcription and Translation¶

Both routes accept multipart/form-data:

curl -s http://localhost:8080/v1/audio/transcriptions \
  -F file=@recording.wav \
  -F model=whisper \
  -F language=ko \
  -F response_format=json

json returns {"text":"..."}. text returns plain UTF-8. verbose_json adds language and duration when available. The translation route uses the same request and response formats but asks Whisper for English output.

Admission Controls¶

Audio work is serialized through one MLX-owning worker thread. These server options bound memory use and wait time:

At the defaults, queued STT payloads can retain roughly 200 MiB plus the in-flight request, because each upload can be 25 MiB.

Errors and Limits¶

Bearer authentication, when configured, applies to audio routes just as it does to other API endpoints.