REST API Reference¶

mlxcel-server exposes generation, audio, embedding, reranking, tokenization, and observability APIs. Generation endpoints support both non-streaming JSON responses and Server-Sent Events (SSE) streaming; embedding and reranking endpoints return ordinary JSON.

Table of Contents¶

• Authentication
• Request Priority
• Embedding and Reranking APIs
• POST /v1/chat/completions
• POST /v1/completions (supports response_format)
• POST /completion (llama-server native completion)
• POST /v1/responses and related (GET, DELETE, cancel)
• POST /v1/messages (Anthropic Messages API)
• OpenAI Audio API
• Realtime VoiceChat WebSocket
• OpenAI Image API
• POST /tokenize
• POST /detokenize
• GET /v1/models
• GET /health
• Prompt Cache Endpoints
• GET /metrics
• GET /slots
• GET /props
• Error Codes

Authentication¶

When the server is started with --api-key <key>, every request except GET /health and GET / must supply the key in the Authorization header:

Authorization: Bearer <key>

For compatibility, the raw token without the Bearer prefix is also accepted in the same header. Requests with a missing or incorrect key receive HTTP 401. The current server authentication middleware checks the Authorization header only; api_key query parameters are not accepted.

When --api-key is not supplied at startup, the server accepts all requests without authentication.

Request Priority¶

The chat-completions, text-completions, Responses, Anthropic Messages, and native /completion generation endpoints accept an optional X-Priority header that controls prefill queue ordering:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Priority: high" \
  -d '{"model": "...", "messages": [...]}'

Embedding and Reranking APIs¶

mlxcel serves OpenAI-compatible POST /v1/embeddings, llama-server-native POST /embedding and /embeddings, and reranking through POST /v1/rerank, /v1/reranking, /rerank, and /reranking. The native embedding paths have a different response shape from the OpenAI path; they are not aliases. Reranking returns a Jina/Cohere envelope for documents or a TEI-compatible bare array for texts.

See Embedding and Reranking APIs for the exact request and response schemas, model-loading forms, flags, errors, and client examples.

POST /v1/chat/completions¶

OpenAI-compatible chat completions. Supports both non-streaming and streaming (SSE) responses. Supports multimodal content parts for VLM/audio-capable models.

Request Body¶

{
  "model": "string",
  "messages": [
    {
      "role": "system | user | assistant | tool",
      "content": "string or array of content parts",
      "name": "string (optional)"
    }
  ],
  "stream": false,
  "stream_options": {
    "include_usage": true
  },
  "max_tokens": 512,
  "temperature": 0.7,
  "top_p": 0.9,
  "top_k": 40,
  "min_p": 0.05,
  "repetition_penalty": 1.1,
  "repetition_context_size": 64,
  "seed": 42,
  "stop": ["string"],
  "frequency_penalty": 0.0,
  "presence_penalty": 0.0,
  "dry_multiplier": 0.0,
  "dry_base": 1.75,
  "dry_allowed_length": 2,
  "dry_penalty_last_n": 0,
  "dry_sequence_breakers": [1, 2, 3],
  "xtc_probability": 0.0,
  "xtc_threshold": 0.1,
  "max_pattern_size": 0,
  "min_pattern_size": 0,
  "min_count": 0
}

Stream Options¶

Tools and tool_choice¶

tools is rendered into the chat template only when the request carries at least one tool; a request without tools, or with "tools": [], leaves the template's tools variable undefined, which is what transformers and llama-server do. tool_choice is enforced rather than echoed:

The instruction goes onto the first system message, otherwise the last user message, otherwise a new leading system message; the request object is not modified. On templates whose tool-call wire shape is a JSON object inside a fixed wrapper (Hermes and Qwen <tool_call>, Mistral Nemo [TOOL_CALLS], Llama 3 <|python_tag|>), a forced choice is also constrained through the structured-output grammar built from the tool schemas, so required yields a call to a declared function and the named form yields that function with finish_reason: "tool_calls". Formats without a JSON wire shape, tool schemas the grammar engine cannot express, and the disaggregated router receive the instruction and the narrowed tool list only, and a forced choice that ends without a call is logged at warn. A forced choice combined with a constraining response_format is rejected with 400. The same rules apply to the Responses API and to the Anthropic Messages {"type": "any"} and {"type": "tool"} mappings.

Speculative decoding timings¶

When a drafter ran at least one verify round for the request, the non-streaming response and the streaming chunk that carries finish_reason include a top-level timings object with llama-server's fields plus draft_n, draft_n_accepted, draft_rounds, and draft_kind (mtp, dflash, or prompt-lookup). For prompt lookup the counters are the drafter's totals for the request. The object is absent on plain decode and on requests that finished inside prefill, and mid-stream chunks never carry the draft counters. The native /completion and /completions routes put the same four fields on their existing timings object. See Speculative Decoding.

Sampling Parameters¶

All sampling parameters are optional. When omitted, server defaults are used (configurable via CLI flags at startup).

Loop detection ends generation early when a short token pattern repeats at the tail of the stream. When a request sets none of the three fields, the server applies its policy: the MLXCEL_LOOP_DETECTION global override if set, otherwise a per-family auto-enable (on by default for the Gemma 4 family, threshold min=1, max=20, count=4). Setting any of the three fields in the request, including an explicit max_pattern_size: 0 to force-disable, always wins over both the global override and the per-family default. See MLXCEL Environment Variables for the operator-level override.

Reproducible greedy output¶

A prompt-cache hit is a split prefill: the cached prefix was evaluated in one pass and the suffix in another, so the first-token logits can differ from a cold run in the last bits, and a near-tie can flip a greedy token. Send "cache_prompt": false when the output must not depend on what earlier requests left in the cache. The native /completion route honors the same field, and --no-cache-prompt is the server-wide form.

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"m","messages":[{"role":"user","content":"Hi"}],"temperature":0,"cache_prompt":false}'

Chat Template Kwargs¶

For conversational models that expose template-level feature switches (such as Qwen3.6's preserve_thinking for multi-turn reasoning retention), per-request kwargs can be supplied in three shapes. They are resolved in this precedence order, with the first match winning:

When chat_template_kwargs is present, unknown keys are forwarded verbatim to the Jinja template, so a template that understands a newer kwarg will pick it up without a server-side allowlist change.

Per-request values merge with the server-wide default set via --chat-template-kwargs / LLAMA_ARG_CHAT_TEMPLATE_KWARGS. Per-request keys win on conflict; unrelated server-default keys persist through the merge.

Primary shape (llama.cpp):

{
  "model": "qwen3.6",
  "messages": [...],
  "chat_template_kwargs": {"preserve_thinking": true}
}

vLLM / OpenAI SDK shape:

{
  "model": "qwen3.6",
  "messages": [...],
  "extra_body": {
    "chat_template_kwargs": {"preserve_thinking": true}
  }
}

# Python OpenAI SDK
client.chat.completions.create(
    model="qwen3.6",
    messages=[...],
    extra_body={"chat_template_kwargs": {"preserve_thinking": True}},
)

DashScope shape (secondary, preserve_thinking only):

{
  "model": "qwen3.6",
  "messages": [...],
  "extra_body": {"preserve_thinking": true}
}

preserve_thinking¶

When true, <think>...</think> blocks from every prior assistant turn are passed to the model unchanged, enabling multi-turn reasoning retention on Qwen3.6. When false (the default), the server applies the Qwen3/Qwen3.5 rolling-checkpoint rule: walk messages in reverse, find the latest non-tool user turn, and strip <think> blocks from every assistant reply before that point.

Non-thinking models (Qwen2, Llama) silently ignore the flag. Qwen3 / Qwen3.5 accept the flag, but the quality benefits of preservation are only validated on Qwen3.6; earlier Qwen3 variants were trained on the rolling-checkpoint convention.

Structured Output (response_format)¶

When response_format is set with type: "json_schema", the server constrains decoding so every emitted token keeps the partial output conforming to the supplied schema. Tokens that would violate the schema are masked to -inf before sampling, so non-conforming output cannot occur.

Request shape (Chat Completions API):

{
  "model": "qwen3-8b",
  "messages": [{"role": "user", "content": "Name one animal and its habitat."}],
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "AnimalResult",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "animal": {"type": "string"},
          "habitat": {
            "type": "string",
            "enum": ["forest", "desert", "ocean", "urban", "unknown"]
          }
        },
        "required": ["animal", "habitat"],
        "additionalProperties": false
      }
    }
  }
}

The json_schema wrapper accepts name and strict alongside schema. These are accepted for OpenAI SDK compatibility but do not change server behavior; the schema field is what drives constrained decoding.

Schema limits, applied before grammar compilation to prevent DoS:

Schemas that exceed any limit are rejected with HTTP 400 and a short descriptive message. The schema content is never echoed in error responses.

Tokenizer requirement: constrained decoding requires a HuggingFace BPE tokenizer (tokenizer.json). Models loaded with a SentencePiece or Tiktoken tokenizer receive HTTP 400 with a UnsupportedTokenizer message when response_format is set.

curl example:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-8b",
    "messages": [{"role": "user", "content": "Name one animal and its habitat."}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "result",
        "schema": {
          "type": "object",
          "properties": {
            "animal": {"type": "string"},
            "habitat": {"type": "string", "enum": ["forest", "desert", "ocean"]}
          },
          "required": ["animal", "habitat"],
          "additionalProperties": false
        }
      }
    }
  }'

Multimodal Content¶

For VLM and audio-capable models, the content field in a message can be an array of content parts instead of a plain string:

{
  "role": "user",
  "content": [
    {
      "type": "text",
      "text": "What is in this image?"
    },
    {
      "type": "image_url",
      "image_url": {
        "url": "data:image/jpeg;base64,/9j/4AAQSkZJRgAB..."
      }
    }
  ]
}

Supported content parts:

The image url field accepts:

• Base64 data URIs: data:image/jpeg;base64,...
• Local file paths: file:///path/to/image.jpg or a bare absolute path
• HTTP/HTTPS URLs: https://example.com/image.jpg

The video url field also accepts data URIs, HTTP/HTTPS URLs, and local file paths. Local video paths are fail-closed by default; set MLXCEL_VIDEO_DIR_ALLOWLIST to a comma-separated list of directories the server may read. The server reads MLXCEL_VIDEO_DIR_ALLOWLIST once at startup, for native-video families and the frame-sampling fallback alike, so restart it after changing the variable or creating a listed directory. Data-video URIs and HTTP(S) URLs are written as create-new 0600 files inside private 0700 directories before the video decoder reads them.

Only Gemma 4 Unified accepts video and audio in the same request; every other family answers 400 with Combined video and audio inputs are not supported. Image+video and image+audio combinations are supported when the loaded model implements the corresponding path. See Video and Audio Inputs for the native-video families and the frames fallback.

Non-Streaming Response¶

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1715000000,
  "model": "qwen3-8b",
  "system_fingerprint": null,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I help you today?"
      },
      "finish_reason": "stop",
      "logprobs": null
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 9,
    "total_tokens": 21
  }
}

The finish_reason field is one of:

Streaming Response (SSE)¶

When "stream": true, the response is a stream of Server-Sent Events. Each event carries a data: field containing a JSON chunk. The stream ends with data: [DONE].

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1715000000,"model":"qwen3-8b","system_fingerprint":null,"choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null,"logprobs":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1715000000,"model":"qwen3-8b","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null,"logprobs":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1715000000,"model":"qwen3-8b","system_fingerprint":null,"choices":[{"index":0,"delta":{},"finish_reason":"stop","logprobs":null}]}

data: [DONE]

When stream_options.include_usage is true, an additional usage chunk is sent after the finish chunk and before the [DONE] marker:

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1715000000,"model":"qwen3-8b","system_fingerprint":null,"choices":[],"usage":{"prompt_tokens":12,"completion_tokens":9,"total_tokens":21}}

data: [DONE]

Tool-call streaming¶

When a tool-call model produces a tool call during streaming, the server's StreamFilter suppresses all tool-call markup and JSON payload from delta.content. Clients see clean content deltas (any prose before the tool call) and then a final chunk with finish_reason: "tool_calls". Model-internal markers never appear in delta.content. Partial delimiter matches at token boundaries are held back until the full marker can be confirmed, then released as content if they turn out not to be a delimiter.

Tool-call names and arguments stream as they are generated for Hermes/Qwen JSON, Mistral ([TOOL_CALLS]NAME[ARGS]{json} and the JSON list form), Llama 3 JSON, and Qwen3-Coder XML. The first chunk of a call carries index, id, type: "function", and function.name; each later chunk carries index and a function.arguments fragment, and concatenating the fragments gives the call's final arguments string. Every other format, and any call the streamer cannot track, is sent as one start chunk plus one arguments chunk at the end of generation. The end-of-generation parse stays authoritative: a streamed call keeps its id, receives the part of its arguments not yet sent, and a disagreement between the stream and the parse is logged at warn with the request id. For these formats an object-valued arguments is the model's own JSON text (key order, spacing, and number forms kept), streamed or not; string-valued arguments are unchanged.

The following tool-call formats are recognized and filtered during streaming. Formats whose bodies are not JSON, such as GLM-4.7, LongCat-Flash and MiniMax M3, coerce argument values against the request's tool schema: a string-typed parameter keeps its raw text, and other types are parsed as JSON or loose literals.

When logprobs is requested, each delimiter token that is suppressed still produces an SSE chunk with an empty delta.content ("") and a populated logprobs field. These placeholder chunks preserve one-to-one correspondence between generated tokens and SSE positions, which is required for correct per-token index alignment when a model emits parallel tool calls. Without this alignment a consumer that indexes logprobs entries by SSE position would silently misattribute probability data across tool-call boundaries.

curl Examples¶

Non-streaming:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-8b",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is 2 + 2?"}
    ],
    "max_tokens": 100,
    "temperature": 0.7
  }'

Streaming:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  --no-buffer \
  -d '{
    "model": "qwen3-8b",
    "messages": [{"role": "user", "content": "Count to 5."}],
    "stream": true
  }'

With image (VLM):

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-vl-7b",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "Describe this image."},
          {"type": "image_url", "image_url": {"url": "file:///tmp/photo.jpg"}}
        ]
      }
    ]
  }'

mlxcel extension fields¶

mlxcel adds a few non-OpenAI fields to its response bodies. Each is optional and omitted unless the loaded model produces it, so a standard client never sees a changed wire shape.

florence2_result carries the structured form of a Florence-2 task answer, so a client does not have to re-parse the formatted string in message.content. The object is {"task": "<OD>", "kind": "bboxes" | "text" | "quad_boxes" | "polygons" | "bboxes_or_polygons", ...} with the coordinate arrays under the upstream mlx-vlm key names (bboxes, quad_boxes, polygons, labels, bboxes_labels, polygons_labels). Coordinates are pixels in the original image extent. Streaming chat responses and /v1/responses return the rendered text only.

reasoning_only is omitted on a response that has content, on a response that ends in tool calls (where empty content is expected), and on an empty generation. It is a separate boolean rather than a new finish_reason value, so clients that validate finish_reason as an enum are unaffected; finish_reason stays length or stop.

POST /v1/completions¶

Legacy text completions endpoint. Accepts a raw prompt string instead of a message array. Suitable for instruction-following without a chat template.

Request Body¶

{
  "model": "string",
  "prompt": "string",
  "stream": false,
  "stream_options": {
    "include_usage": true
  },
  "max_tokens": 512,
  "temperature": 0.7,
  "top_p": 0.9,
  "top_k": 40,
  "min_p": 0.05,
  "repetition_penalty": 1.1,
  "seed": null,
  "stop": ["string"],
  "frequency_penalty": 0.0,
  "presence_penalty": 0.0,
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "result",
      "schema": {"type": "object", "properties": {"value": {"type": "string"}}, "required": ["value"]}
    }
  }
}

Sampling parameters are identical to /v1/chat/completions. The response_format field follows the same shape described in the Structured Output section above: type: "json_schema" enables constrained decoding; type: "text" or absent means normal generation; type: "json_object" is rejected with HTTP 400.

Non-Streaming Response¶

{
  "id": "cmpl-abc123",
  "object": "text_completion",
  "created": 1715000000,
  "model": "qwen3-8b",
  "system_fingerprint": null,
  "choices": [
    {
      "index": 0,
      "text": "The capital of France is Paris.",
      "finish_reason": "stop",
      "logprobs": null
    }
  ],
  "usage": {
    "prompt_tokens": 8,
    "completion_tokens": 7,
    "total_tokens": 15
  }
}

Streaming Response¶

Same SSE format as /v1/chat/completions. The delta field contains "text" instead of "content".

curl Example¶

curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-8b",
    "prompt": "The capital of France is",
    "max_tokens": 20,
    "temperature": 0
  }'

OpenAI Responses API (/v1/responses)¶

mlxcel exposes the OpenAI Responses API surface as a dedicated endpoint family. The Python SDK's client.responses.create(...) works against mlxcel-server without a chat-completions shim.

Endpoints¶

Aliases without the /v1 prefix (/responses, /responses/{id}, /responses/{id}/cancel, /responses/compact, /responses/input_tokens) are also mounted.

Quick example¶

curl http://127.0.0.1:8080/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3-0.6b-4bit",
    "input": "Reply with: hello",
    "max_output_tokens": 64,
    "thinking_budget_tokens": 0
  }'

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="sk-no-auth")
resp = client.responses.create(
    model="qwen3-0.6b-4bit",
    input="Reply with: hello",
    max_output_tokens=64,
)
print(resp.output_text)

Supported surface¶

The /v1/responses endpoint supports text input (string or message item arrays), message content parts that reuse the chat-completions multimodal schema, function tools, JSON-schema structured output via text.format, previous_response_id chaining, the string-form conversation field, and streamed responses with typed SSE events that carry a monotonic sequence_number.

Built-in tool types (web_search, file_search, computer_use_preview, code_interpreter, image_generation, mcp, custom, apply_patch, function_shell) return 400 invalid_request_error. input_image content parts are accepted in messages, but input_file parts, truncation other than "disabled", and background: true return 400.

An input item without type that carries a role is read as a message, and assistant messages accept output_text and refusal content parts, so a client can replay a previous response.output as input. On templates whose generation prompt already opens a reasoning block (ending in <think>\n), the reasoning streams as response.reasoning_text.delta events in a reasoning item ahead of the message.

Compaction¶

POST /v1/responses/compact takes the full history in input and returns {"id", "object": "response.compaction", "created_at", "model", "output", "usage"}. output holds the kept system and developer messages, the retained items, and one compaction item whose encrypted_content is a base64url capsule (not encrypted; it carries the summary and the retained items). Nothing is stored. The retained part is the latest user message plus the newest whole turns within keep_tokens (default 1/16 of the context window, at most 8192); splits fall only before a user turn or after a completed tool group, so a function call is never separated from its output. The summary is one non-streaming generation, bounded by max_output_tokens (default 1024, 1 to 16384). stream, previous_response_id, and conversation are rejected, and keep_tokens must be above 0. A history longer than one context window, a summary that does not fit, or a result that is not shorter than the input returns 400 with code context_length_exceeded and changes nothing.

/v1/responses and /v1/responses/input_tokens accept a compaction input item and replay it, including through previous_response_id chains.

/v1/responses also accepts context_management: [{"type": "compaction", "compact_threshold": N}] (N from 1 to 100000000). When the rendered input, counted as /v1/responses/input_tokens counts it, reaches N or leaves no room for output, the history is compacted as above before generation and output starts with the compaction item. Streamed requests send mlx.compaction.started and mlx.compaction.completed (with input_tokens_before and input_tokens_after) after response.in_progress. For a request with context_management, generation is capped at the context left after the prompt, and a length stop returns status: "incomplete" with reason max_output_tokens.

CLI flags¶

Set the store cap to 0 to disable persistence entirely; in that mode GET/DELETE/cancel and previous_response_id all return 400.

For the full request/response field reference, streaming-event catalog, and migration guide from /v1/chat/completions, see OpenAI Responses API.

Anthropic Messages API (/v1/messages)¶

mlxcel exposes the Anthropic Messages API so the official Anthropic SDKs (client.messages.create(...)) work against mlxcel-server without a chat-completions shim.

Endpoints¶

Aliases without the /v1 prefix (/messages, /messages/count_tokens) are also mounted.

Quick example¶

curl http://127.0.0.1:8080/v1/messages \
  -H 'Content-Type: application/json' \
  -H 'anthropic-version: 2023-06-01' \
  -d '{
    "model": "qwen3-0.6b-4bit",
    "max_tokens": 64,
    "system": "Be terse.",
    "messages": [{"role": "user", "content": "Reply with: hello"}]
  }'

from anthropic import Anthropic

client = Anthropic(base_url="http://127.0.0.1:8080", api_key="sk-no-auth")
msg = client.messages.create(
    model="qwen3-0.6b-4bit",
    max_tokens=64,
    messages=[{"role": "user", "content": "Reply with: hello"}],
)
print(msg.content[0].text)

Supported surface¶

Top-level system (string or text-block array), messages, max_tokens, stop_sequences, temperature/top_p/top_k, function tools + tool_choice, and thinking are honoured. Content blocks support text, image (base64 and url sources), tool_use, and tool_result, including image content inside tool_result blocks, so vision models can consume tool-produced images. Streaming emits the Anthropic SSE event sequence (message_start, content_block_start/delta/stop, message_delta, message_stop) and usage token counts are populated on both the non-streaming response and the streaming message_delta.

Anthropic server tools (web_search, code_execution, and other tools without an input_schema) are accepted but dropped, because a local model server cannot execute them. Unknown top-level fields (metadata, container, service_tier, …) are accepted and ignored.

With extended thinking enabled, templates whose generation prompt already opens a reasoning block return that reasoning as a thinking content block.

For the full request/response field reference, the content-block mapping (including image-in-tool-result), the streaming-event catalog, and the token-counting endpoint, see Anthropic Messages API.

POST /completion¶

llama-server-compatible native completion. Use this endpoint when migrating a llama.cpp client that expects content, stop, tokens_predicted, tokens_evaluated, and timings in the response.

curl http://localhost:8080/completion \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"The capital of France is","n_predict":32,"temperature":0,"stream":false}'

The request accepts prompt, n_predict, stream, temperature, top_k, top_p, min_p, repeat_penalty, repeat_last_n, stop, seed, frequency/presence penalties, DRY settings, p_less, cache_prompt, and the three thinking-budget aliases. Structured response_format is not supported here and returns 400 except for the no-op {"type":"text"} form; use /v1/chat/completions for JSON-schema constrained decoding.

OpenAI Audio API (/v1/audio)¶

The server mounts POST /v1/audio/speech, /v1/audio/transcriptions, and /v1/audio/translations, plus aliases without /v1. Whisper and Kokoro run in audio-only server processes. See OpenAI Audio API for checkpoint setup, multipart/JSON schemas, WAV formats, 25 MiB upload limits, queue controls, and errors.

Realtime VoiceChat WebSocket (/v1/realtime)¶

When the server loads a Nemotron VoiceChat checkpoint, it mounts GET /v1/realtime as a WebSocket upgrade for full-duplex speech and answers the chat and completion routes with 501. See Realtime VoiceChat API for the session protocol, events, and limits.

OpenAI Image API (/v1/images)¶

When the server is started with --image-model (or MLXCEL_IMAGE_MODEL), it mounts POST /v1/images/generations and POST /v1/images/edits, plus the aliases /images/generations and /images/edits. The image worker is loaded at startup like the embedding worker, runs one request at a time, answers 503 when its queue is full, and 504 on a per-request timeout. Without the flag both routes answer 501 and name it. Bodies are JSON only; multipart upload is not served. response_format accepts only b64_json and output_format only png; url, path, output_path, and output_dir return 400, because the server writes no files for a client. A successful response is {"created", "data": [{"b64_json", "mime_type", "width", "height", "seed", "revised_prompt"}], "output_format", "size"}. See Image Generation for the checkpoint families, the request fields, and the error split.

POST /tokenize¶

Tokenize a text string using the loaded model's tokenizer. Useful for counting tokens before submitting a generation request.

Request Body¶

{
  "content": "Hello, world!",
  "add_special": false
}

Response¶

{
  "tokens": [9906, 11, 1917, 0]
}

The tokens array contains signed 32-bit integer token IDs.

curl Example¶

curl http://localhost:8080/tokenize \
  -H "Content-Type: application/json" \
  -d '{"content": "Hello, world!", "add_special": false}'

POST /detokenize¶

Convert a sequence of token IDs back to a text string.

Request Body¶

{
  "tokens": [9906, 11, 1917, 0]
}

Response¶

{
  "content": "Hello, world!"
}

curl Example¶

curl http://localhost:8080/detokenize \
  -H "Content-Type: application/json" \
  -d '{"tokens": [9906, 11, 1917, 0]}'

A negative id or an id outside the vocabulary returns 500 with type server_error and the message token id {id} is out of vocabulary range (vocab_size {n}), naming the first bad id in request order. The check runs before decoding, so no partial text is returned. This matches llama-server's status and error type.

GET /v1/models¶

List the models available on this server instance. mlxcel loads a single model at startup, so this endpoint always returns a list with one entry.

Response¶

{
  "object": "list",
  "data": [
    {
      "id": "qwen3-8b",
      "object": "model",
      "created": 1715000000,
      "owned_by": "user"
    }
  ]
}

curl Example¶

curl http://localhost:8080/v1/models

GET /health¶

Health check endpoint. GET / is an unauthenticated alias with the same handler. Both paths remain accessible when an API key is configured. The response reports server liveness, model loading status, current batch metrics, context window size, and tool-call parser availability.

Response¶

Model loaded and capacity available (HTTP 200):

{
  "status": "ok",
  "model": "qwen3-8b",
  "batch": {
    "active_sequences": 2,
    "queue_depth": 0,
    "max_batch_size": 4
  },
  "observability": {
    "sequences_started": 42,
    "sequences_completed": 40,
    "total_prefill_tokens": 8192,
    "total_decode_tokens": 4096,
    "decode_steps_processed": 256,
    "prefill_chunks_processed": 42,
    "current_batch_size": 2,
    "cache_pool_active": 2
  },
  "context_size": 4096,
  "tool_call_parser": "mlxcel"
}

All slots busy (HTTP 503):

{
  "status": "no slot available",
  "model": "qwen3-8b",
  "batch": { ... },
  "observability": { ... },
  "context_size": 4096,
  "tool_call_parser": "mlxcel"
}

Model still loading (HTTP 503):

{
  "status": "loading model",
  "tool_call_parser": null
}

Response Fields¶

curl Example¶

curl -i http://localhost:8080/health

Prompt Cache Endpoints¶

Prompt-cache observability and reset endpoints are always mounted. When the prompt cache is disabled, they return a stable disabled payload so monitoring clients do not need conditional routing.

GET /v1/cache/stats¶

Returns the current prompt-cache snapshot:

{
  "enabled": true,
  "apc_enabled": false,
  "block_size": 16,
  "hash": "sha256",
  "entries": 12,
  "bytes": 104857600,
  "capacity_bytes": 2147483648,
  "max_entries": 1024,
  "hits": 42,
  "lookups": 60,
  "hit_rate": 0.7,
  "inserts": 18,
  "evictions_lru": 0,
  "evictions_ttl": 0,
  "rejections_oversized": 0,
  "total_blocks_stored": 0,
  "unique_block_hashes": 0,
  "apc_active_entries": 0
}

POST /v1/cache/reset¶

Clears all live prompt-cache entries and reports the freed snapshot. The call is idempotent.

{
  "cleared": true,
  "freed_bytes": 104857600,
  "freed_entries": 12
}

GET /metrics¶

Prometheus-compatible metrics in text format. Enabled when the server is started with --metrics or --metrics-port. There is no /metrics route unless one of those flags is supplied.

mlxcel_requests_total counts completed generation requests. A chat completion counts once whether it streams or not; a streamed one is recorded when its generation finishes, and a chat completion stream the client cancelled is excluded. mlxcel_completion_tokens_total sums the completion tokens of the requests counted there. Streaming /v1/completions and native /completion requests do not record.

Response¶

Content-Type: text/plain; version=0.0.4; charset=utf-8

# HELP mlxcel_requests_total Total number of completed generation requests; chat completion streams cancelled by the client are excluded
# TYPE mlxcel_requests_total counter
mlxcel_requests_total 42

# HELP mlxcel_prompt_tokens_total Total prompt tokens processed
# TYPE mlxcel_prompt_tokens_total counter
mlxcel_prompt_tokens_total 8192

# HELP mlxcel_completion_tokens_total Total completion tokens of the requests counted in mlxcel_requests_total
# TYPE mlxcel_completion_tokens_total counter
mlxcel_completion_tokens_total 4096

# HELP mlxcel_generation_time_seconds_total Total generation time in seconds
# TYPE mlxcel_generation_time_seconds_total counter
mlxcel_generation_time_seconds_total 12.345

# HELP mlxcel_slots_total Total number of parallel slots
# TYPE mlxcel_slots_total gauge
mlxcel_slots_total 4

# HELP mlxcel_slots_available Available parallel slots
# TYPE mlxcel_slots_available gauge
mlxcel_slots_available 2

# HELP mlxcel_queue_depth Current prefill queue depth
# TYPE mlxcel_queue_depth gauge
mlxcel_queue_depth 0

# HELP mlxcel_batch_sequences_started Total sequences that entered prefill
# TYPE mlxcel_batch_sequences_started counter
mlxcel_batch_sequences_started 42

# HELP mlxcel_batch_sequences_completed Total sequences that completed generation
# TYPE mlxcel_batch_sequences_completed counter
mlxcel_batch_sequences_completed 40

# HELP mlxcel_batch_prefill_tokens_total Cumulative prefill tokens processed
# TYPE mlxcel_batch_prefill_tokens_total counter
mlxcel_batch_prefill_tokens_total 8192

# HELP mlxcel_batch_decode_tokens_total Cumulative decode tokens generated
# TYPE mlxcel_batch_decode_tokens_total counter
mlxcel_batch_decode_tokens_total 4096

# HELP mlxcel_batch_current_size Current active batch size
# TYPE mlxcel_batch_current_size gauge
mlxcel_batch_current_size 2

# HELP mlxcel_cache_pool_active Active cache entries
# TYPE mlxcel_cache_pool_active gauge
mlxcel_cache_pool_active 2

curl Example¶

curl http://localhost:8080/metrics

GET /slots¶

Per-slot status endpoint. Enabled when the server is started with --slots (or when the feature is not explicitly disabled). Reports the state of each decode slot and any queued requests.

Response¶

[
  {
    "id": 0,
    "state": "decoding",
    "model": "qwen3-8b",
    "context_size": 8192,
    "is_processing": true
  },
  {
    "id": 1,
    "state": "idle",
    "model": "qwen3-8b",
    "context_size": 8192,
    "is_processing": false
  },
  {
    "id": 2,
    "state": "queued",
    "model": "qwen3-8b",
    "context_size": 8192,
    "is_processing": false
  }
]

context_size is the same effective per-slot value reported by /health.

Slot states:

curl Example¶

curl http://localhost:8080/slots

GET /props¶

Server properties endpoint. Returns the current default generation settings and total slot count. Compatible with the llama-server /props format.

Response¶

{
  "default_generation_settings": {
    "n_predict": 512,
    "temperature": 0.7,
    "top_k": 40,
    "top_p": 0.9,
    "min_p": 0.05,
    "repeat_penalty": 1.1,
    "repeat_last_n": 64,
    "seed": 18446744073709551615,
    "frequency_penalty": 0.0,
    "presence_penalty": 0.0,
    "dry_multiplier": 0.0,
    "dry_base": 1.75,
    "dry_allowed_length": 2,
    "dry_penalty_last_n": 0
  },
  "total_slots": 4
}

The seed field defaults to u64::MAX (18446744073709551615) when no seed is configured, which means non-deterministic sampling.

curl Example¶

curl http://localhost:8080/props

Error Codes¶

All error responses follow the OpenAI error format:

{
  "error": {
    "message": "human-readable description",
    "type": "error_type_string",
    "code": null
  }
}

Example 503 response:

{
  "error": {
    "message": "All slots are busy. Please try again later.",
    "type": "server_busy",
    "code": null
  }
}

Admission Control¶

mlxcel applies queue-depth admission control on all generation endpoints. Before opening an SSE stream or running a generation, the server checks whether there is capacity in the prefill queue. If not, it returns HTTP 503 immediately rather than accepting the connection and hanging.

This means clients should treat 503 responses as a backpressure signal and implement retry logic with exponential backoff.

Request memory admission (Metal Nemotron-H)¶

On Metal, a text-only Nemotron-H checkpoint runs with one active slot, and each request is admitted against a conservative estimate of its memory use. A request whose estimate exceeds the budget gets 503 with type and code set to request_memory_budget_exceeded and the extra fields estimated_bytes and budget_bytes. Streaming requests are refused the same way, before any SSE headers are sent. The budget is the smallest of the device's recommended working set, the configured wired-memory cap, and the MLX allocator cap. MLXCEL_REQUEST_MEMORY_BUDGET (a byte size such as 12G) can only lower it; zero, none, and other invalid values stop startup.

{
  "error": {
    "message": "request_memory_budget_exceeded: estimated_bytes=..., budget_bytes=...; shorten the prompt/output or lower retained cache usage",
    "type": "request_memory_budget_exceeded",
    "code": "request_memory_budget_exceeded",
    "estimated_bytes": 2048,
    "budget_bytes": 1024
  }
}