Realtime VoiceChat API¶
NemotronLabs VoiceChat (model_type: "nemotron_voicechat", architectures: ["NemotronVoiceChatForConditionalGeneration"]) is a full-duplex speech-to-speech model. It consumes 16 kHz audio in 80 ms frames and, on every frame, emits an assistant text token, a function-channel token, a user-transcript update, and 80 ms of 22.05 kHz assistant speech, all on one shared timeline. Silence and overlapping speech are part of the timeline; there is no voice-activity gate.
mlxcel runs it offline from mlxcel generate and live over the /v1/realtime WebSocket of mlxcel serve and mlxcel-server. The protocol follows the mlx-vlm reference server (mlx_vlm/server/realtime.py).
Checkpoints¶
| Checkpoint | Notes |
|---|---|
mlx-community/NemotronLabs-VoiceChat-11B-bf16 | Unquantized |
mlx-community/NemotronLabs-VoiceChat-11B-8bit | LLM quantized (embedding table, both heads, Nemotron-H mixer projections) |
mlx-community/NemotronLabs-VoiceChat-11B-4bit | Same quantization scope as 8-bit |
In the quantized conversions the speech encoder, the RNNT transcription branch, the EAR-TTS speech generator, and the codec stay bf16. Only the converted MLX safetensors layout loads; the original NeMo .nemo checkpoint does not.
How a Frame Is Produced¶
| Network | What it does |
|---|---|
| Log-mel frontend | 25 ms / 10 ms STFT, 128 mel bins. |
| FastConformer encoder | 8x causal subsampling (10 ms mel frames become 80 ms frames) and 24 conformer blocks with 70 frames of left attention context. |
| RNNT branch | Greedy transcription of the user audio. |
| Duplex LLM | Nemotron-H (56 layers) over one fused embedding per position (previous text token, audio, previous function token), with greedy text and function heads. |
| EAR-TTS | Gemma-3-style backbone conditioned on the current text token; produces 31 residual codebook codes per frame. |
| Codec | Decodes one code frame into 1764 samples (80 ms at 22.05 kHz). |
Offline CLI¶
mlxcel generate -m models/NemotronLabs-VoiceChat-11B-4bit \
--audio question.wav --output-audio response.wav \
-p "Be concise and answer in one sentence."
The command prints [user] <transcript>, the assistant text, and [function] <text> when the function channel produced any, and writes response.wav as 22050 Hz mono PCM16 covering the whole timeline.
| Option | Behavior for a VoiceChat checkpoint |
|---|---|
--audio <PATH> | Required. The user side of the duplex timeline, resampled to 16 kHz mono. |
-p, --prompt | Optional system prompt; omitted means no system prompt. |
--output-audio <PATH> | Writes the assistant speech (22050 Hz mono PCM16). |
--extra-decoding-seconds <S> | Seconds of silence appended to the input so the model can finish answering. Default 3.0. |
--seed <N> | Seeds the EAR-TTS sampling noise. Default 0; two runs with the same seed produce byte-identical audio. |
--stream | Drives the cache-aware online session in 80 ms frames instead of the offline timeline, printing assistant text as frames produce it. |
--max-streaming-seconds <S> | With --stream, stops with an error once the session has consumed this many seconds of audio. |
--profile | With --stream, prints a per-stage latency summary and the real-time factor after dropping 5 cold frames. |
-n | Ignored; the output length equals the input timeline. |
--image, --video | Rejected. |
Interactive mlxcel generate without --audio refuses the checkpoint with a pointer to the offline command.
Streamed Profile¶
--stream --profile reports, per stage (perception, rnnt, language, tts, codec, total), mean, p50, p95, and max frame time, plus realtime_factor = mean total / 80 ms. A value above 1.0 is slower than real time. Measured on an idle Apple M1 Ultra (128 GB) with a short question plus 3 s of extra decoding:
| Checkpoint | Real-time factor | Frame time p50 / p95 |
|---|---|---|
| 4-bit | 0.87 | 69.0 ms / 70.4 ms |
| 8-bit | 0.94 | 75.3 ms / 76.4 ms |
The loader holds the encoder, the EAR-TTS backbone, and the EAR-TTS mixture-of-Gaussians head as f32 (exact conversion from bf16), which costs about 2.7 GB of extra resident memory and avoids a per-frame weight cast.
Two diagnostic variables refine the profile:
| Variable | Effect |
|---|---|
MLXCEL_VOICECHAT_PROFILE_STAGES=1 | Adds a sub_stages map (for example perception.layers, tts.backbone, codec.decoder_istft). The extra timers force evaluation at every boundary, so that run's frame totals are not the real-time factor. |
MLXCEL_VOICECHAT_PROFILE_FRAMES=<path> | Writes every frame's timings, cold frames included, as JSON. |
Serving¶
mlxcel-server -m models/NemotronLabs-VoiceChat-11B-4bit --port 8080
# or
mlxcel serve --model models/NemotronLabs-VoiceChat-11B-4bit --port 8080
The server loads the checkpoint on a dedicated realtime engine thread and mounts GET /v1/realtime as a WebSocket upgrade. The chat worker, the batch scheduler, and the text warmup do not start for this checkpoint. /v1/chat/completions, /v1/completions, /v1/responses, and /v1/messages answer 501 Not Implemented with a message naming /v1/realtime.
| Server option | Effect on /v1/realtime |
|---|---|
--api-key | The upgrade request needs Authorization: Bearer <key>. |
--api-prefix | Prefixes the path like every other route. |
--cors-origins, --allowed-origins | Gate the upgrade by its Origin header. A browser origin the policy does not allow gets 403 Forbidden; a client that sends no Origin (a non-browser client) is accepted. |
--alias | The model id reported in session.updated; otherwise the model directory name. |
Limits¶
| Limit | Value |
|---|---|
| Concurrent sessions | One. A second connection while a session is active receives server_busy and close code 1013 (try again later). |
| WebSocket message or frame size | 1 MiB (about 24 s of base64 PCM16; clients normally send 80 ms appends). |
| Audio per session | 600 s. session.max_streaming_seconds can lower it; larger values are clamped. Reaching the limit ends the stream with inference_error. |
Time to send session.update | 30 s after connecting. |
| Idle timeout | 120 s with no client message on a configured session (WebSocket pings do not count); the server sends invalid_request and closes with code 1008. |
| Voice | Only the checkpoint's built-in Aria voice; any other speaker is rejected at load. |
| Batch size | 1. |
The session slot is released when the session ends: after commit or cancel, and when the client disconnects, in which case the server cancels the unflushed session first.
Protocol¶
All messages are JSON text frames.
Client Messages¶
type | Fields | Behavior |
|---|---|---|
session.update | session.system_prompt (string; absent uses the checkpoint default, which is empty), session.seed (non-negative integer, default 0), session.max_streaming_seconds (number, optional), session.model (optional; any value opens the served checkpoint) | Opens the streaming session (warms EAR-TTS and prefills the system prompt) and is answered with session.updated. A second session.update on a configured connection is an invalid_request error. |
input_audio_buffer.append | audio (base64 little-endian PCM16 mono), sample_rate (default 16000; any other rate is rejected) | Pushed in 1280-sample (80 ms) slices. The events of each slice are sent before the next slice runs. |
input_audio_buffer.commit | pad_partial (default true) | Answered with input_audio_buffer.committed. The session is flushed (the partial frame is zero-padded when pad_partial is true), its events end with response.done, and the server closes the socket with code 1000. |
session.cancel, response.cancel | Emits response.cancelled and closes the socket with code 1000. | |
session.ping | Answered with session.pong. |
Before session.update, every message other than session.update and session.ping gets an invalid_request error ("send session.update before audio"). After it, an unknown type is an invalid_request error. A frame that is not a JSON object is an invalid_request error and the connection stays open.
Server Events¶
Every event carries event_id (event_ plus 16 hex digits).
type | Fields |
|---|---|
session.created | session.id (sess_ plus 16 hex digits), session.state: "configuring", session.input_audio_format ({"type": "pcm16", "sample_rate": 16000}), session.output_audio_format ({"type": "pcm16", "sample_rate": 22050}) |
session.updated | session.id, session.state: "ready", session.model, session.frame_samples: 1280, both audio formats |
conversation.item.input_audio_transcription.delta | frame_index, delta, cumulative transcript |
response.text.delta | frame_index, token_id, delta, cumulative text |
response.function.delta | frame_index, token_id, delta, cumulative text |
response.audio.delta | frame_index, delta (base64 PCM16 of round(clip(x, -1, 1) * 32767), 1764 samples), format: "pcm16", sample_rate: 22050, channels: 1, audio_codes (the frame's 31 codec codes) |
input_audio_buffer.committed | |
response.done, response.cancelled | frame_index |
session.pong | |
error | error.code, error.message |
The cumulative text and transcript fields are authoritative; delta is the new suffix when the cumulative decode extends the previous one.
Error Codes¶
error.code | When |
|---|---|
invalid_request | Unknown type, a frame that is not a JSON object, audio before session.update, a second session.update, or the idle timeout. |
server_busy | Another connection holds the session. The socket closes with code 1013. |
session_initialization_failed | session.update failed, for example on a negative seed or a non-positive max_streaming_seconds. The connection stays open and may retry. |
inference_error | An append whose audio is not valid base64 or has an odd byte count, a non-integer sample_rate, a rate other than 16 kHz, a stream past max_streaming_seconds, or a model failure. The connection stays open; a failed append stops at the failing slice. |
Example Exchange¶
Client messages are prefixed with >; event_id values and audio payloads are elided.
{"type":"session.created","session":{"id":"sess_…","state":"configuring","input_audio_format":{"type":"pcm16","sample_rate":16000},"output_audio_format":{"type":"pcm16","sample_rate":22050}}}
> {"type":"session.update","session":{"system_prompt":"Be concise and answer in one sentence.","seed":0}}
{"type":"session.updated","session":{"id":"sess_…","state":"ready","model":"nemotronlabs-voicechat-11b-4bit","frame_samples":1280,"input_audio_format":{"type":"pcm16","sample_rate":16000},"output_audio_format":{"type":"pcm16","sample_rate":22050}}}
> {"type":"input_audio_buffer.append","audio":"…","sample_rate":16000}
{"type":"response.audio.delta","frame_index":0,"delta":"…","format":"pcm16","sample_rate":22050,"channels":1,"audio_codes":[…]}
{"type":"conversation.item.input_audio_transcription.delta","frame_index":5,"delta":"What","transcript":"What"}
{"type":"response.text.delta","frame_index":24,"token_id":…,"delta":"The","text":"The"}
> {"type":"input_audio_buffer.commit"}
{"type":"input_audio_buffer.committed"}
{"type":"response.done","frame_index":61}
A session computes the same result as mlxcel generate --stream for the same input, system prompt, and seed. The audio files are not byte-identical because the CLI's WAV writer scales samples by 32768 while the wire format scales by 32767, so each sample may differ by one quantization step.
The model keeps listening while it speaks and the server performs no acoustic echo cancellation. A live microphone client should play the reply through headphones; through open speakers the model hears its own answer.