Server Mode¶

Table of Contents¶

• What Is Server Mode?
• Starting the Server
• Authentication
• Media inputs
• Endpoint Overview
• Serving Audio Models
• Serving Nemotron VoiceChat (Full-Duplex Speech)
• Serving Embedding and Reranking Models
• Server Lifecycle
• Monitoring
• Integration Examples
• Long Prefill and Proxy Timeouts
• KV Cache Size Limit (--max-kv-size)
• Nemotron-H Request Memory Admission
• Loop Detection (N-gram Repetition Guard)
• Reasoning Re-echo for Content-Only Clients
• Thinking-Token Budget (Qwen3 family)
• Chat Template Kwargs
• Serving Florence-2
• Serving Block-Diffusion Models

What Is Server Mode?¶

Server mode turns mlxcel into a persistent HTTP service. Instead of loading the model for each request and exiting (as mlxcel generate does), the server loads the model once, keeps it resident in memory, and serves an unlimited number of inference requests over its lifetime.

This design is appropriate whenever:

• Multiple clients need access to the same model simultaneously.
• Request latency matters and the model load time (which can be several seconds for large models) is unacceptable per call.
• You want a drop-in replacement for the OpenAI API or a llama.cpp server.

The server exposes OpenAI-compatible generation, Responses, Audio, and Embeddings; Anthropic Messages; Jina/Cohere/TEI-compatible reranking; and llama-server-native completion, embedding, tokenization, and observability routes. Clients that already use those contracts normally need only a base URL change, but native and OpenAI embedding/completion paths deliberately have different response shapes.

Starting the Server¶

Two binaries can start the server:

Both binaries share the same startup logic in src/server/startup.rs. Most runtime flags are shared, but a few spellings intentionally differ for llama.cpp compatibility: mlxcel serve presents --n-parallel and --draft-model as its primary spellings, while standalone mlxcel-server presents --parallel and --model-draft; the corresponding alternate spellings are aliases. See Speculative Decoding and CLI Reference - mlxcel serve.

Minimal startup:

# Bare names resolve to mlx-community/<name> and auto-download on first use.
mlxcel serve --model Qwen2.5-7B-Instruct-4bit

Standalone server binary:

mlxcel-server --model Qwen2.5-7B-Instruct-4bit --port 8080

The model argument accepts an existing local path, a HuggingFace owner/name repo-id, or a bare name resolved through ${MLXCEL_DEFAULT_ORG:-mlx-community}. Repo-id misses are downloaded into the mlxcel global store. Use --models-dir <PATH> or MLXCEL_MODELS_DIR to relocate that store for serve and mlxcel-server.

The server defaults to four parallel request slots (--parallel 4 / --n-parallel 4) and automatic decode storage. Model families that cannot batch are clamped to one slot. Prompt-prefix caching and APC are enabled by default; use --no-prompt-cache or --apc-enabled=false for controlled comparisons.

DFlash speculative decoding server:

mlxcel-server \
  --model ./models/qwen3.5-27b-4bit \
  --model-draft ./models/Qwen3.5-27B-DFlash \
  --draft-kind dflash \
  --draft-block-size 16 \
  --port 8080

For mlxcel serve, the primary spelling for the same drafter path is --draft-model (--model-draft is also accepted):

mlxcel serve \
  --model ./models/qwen3.5-27b-4bit \
  --draft-model ./models/Qwen3.5-27B-DFlash \
  --draft-kind dflash \
  --draft-block-size 16 \
  --port 8080

Full-model drafter (a smaller checkpoint with the target's tokenizer):

mlxcel serve \
  --model ./models/Qwen3-8B-4bit \
  --draft-model ./models/Qwen3-0.6B-4bit \
  --draft-kind full-model \
  --port 8080

The worker loads the drafter next to the target at startup. Each eligible request drafts on its own draft caches and runs one verify round per decode tick while the active batch has at most --prompt-lookup-max-batch rows (default 2).

Prompt-lookup speculation (no drafter checkpoint):

mlxcel serve \
  --model ./models/Qwen3-1.7B-4bit \
  --draft-kind prompt-lookup \
  --port 8080

Prompt lookup proposes tokens copied from earlier in the context and verifies them inside the batch scheduler. It is opt-in, pays off on edit-style requests, and runs verify rounds only while the active batch has at most --prompt-lookup-max-batch rows (default 2). See Speculative Decoding for eligibility and measurements. At temperature 0, prompt-lookup and full-model rounds can leave the plain-decode token stream, so their output is not guaranteed to match a run without speculation (#2778).

Binding to all interfaces (for Docker or remote access):

mlxcel serve \
  --model ./models/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8080

Unix socket mode:

A --host value ending in .sock is served as a Unix domain socket at that path. --port is ignored in that case:

mlxcel serve \
  --model ./models/Gemma-3-4B-Instruct \
  --host /var/run/mlxcel.sock

Unix socket mode is useful when the server and client run on the same host and you want to avoid TCP overhead and open network ports.

--port 0 is a TCP setting: it binds an ephemeral port chosen by the kernel.

Bundled WebUI:

--webui serves the administrative browser UI at /webui/ on the same listener. Without a model argument the server starts model-free, and models are loaded from the UI:

mlxcel-server --webui --models-dir ./models/mlx --port 8080

On a loopback listener with no --api-key, the server prints a one-time session key to the terminal. A non-loopback listener requires an explicit API key and TLS, and a Unix socket listener is refused. See WebUI.

Authentication¶

By default, the server accepts requests without any authentication. This is suitable for local development but should not be exposed to untrusted networks without setting an API key.

When an API key is configured, all endpoints except /health and / require the key to be passed as a Bearer token in the Authorization header.

Media inputs¶

A file:// media URL is refused outright unless the server was started with --media-path <dir>. With a root configured, a request's path resolves relative to that root first and an absolute path is accepted only when it names a file inside it. Under --media-path /srv/media, file://cat.png, file:///srv/media/cat.png and a bare cat.png all read /srv/media/cat.png; file:///etc/passwd is refused as an escape. The resolved path is canonicalized and must stay inside the canonical root, so a symlink pointing out of it is refused while one staying inside still resolves. The open uses O_NOFOLLOW and accepts regular files only, so neither a last-component symlink swap nor a FIFO under the root can be used against the server. Paths carrying .., control characters, : * ? " < > |, a leading or trailing space, a trailing ., more than 255 bytes, Unicode separator look-alikes, or a percent escape for ., /, \ or NUL are rejected before any of that.

An http(s) media URL is checked for a non-public address in three places: before the request (scheme, no credentials in the URL, and every address a hostname resolves to), on every redirect in the five-hop chain, and again on the peer address actually reached before a single body byte is read, which closes the DNS-rebinding window. Loopback, the unspecified address, multicast and broadcast, the IPv4 private and link-local ranges (which puts the cloud metadata service at 169.254.169.254 out of reach), carrier-grade NAT, the protocol-assignment, benchmarking and documentation prefixes, 240.0.0.0/4, IPv6 unique-local, link-local and documentation, and any IPv4-mapped or IPv4-compatible IPv6 spelling of a refused IPv4 address are all refused. A deployment serving media from an internal object store sets MLXCEL_ALLOW_PRIVATE_MEDIA_URLS=1, which turns every address check into a pass.

Payload limits are unchanged by either policy: --max-image-payload-size, --max-images, the decoder's width, height and allocation caps, the audio per-clip and per-request ceilings, and the 10-second fetch timeout all still apply.

A video_url block sent to a checkpoint with a vision tower but no native video path is decoded and sent to the model as ordered still images. --video-max-frames (default 16, MLXCEL_VIDEO_MAX_FRAMES) caps the kept frames and --video-fps (default 2.0, MLXCEL_VIDEO_FPS) sets the sampling rate when the block carries no fps. The frames count against --max-images. See Video on a model with no native video path.

Setting the key inline:

mlxcel serve \
  --model ./models/Qwen2.5-7B-Instruct \
  --api-key "sk-my-secret-key"

Reading the key from a file:

echo "sk-my-secret-key" > /etc/mlxcel/api-key
mlxcel serve \
  --model ./models/Qwen2.5-7B-Instruct \
  --api-key-file /etc/mlxcel/api-key

Sending an authenticated request:

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer sk-my-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen2.5-7B-Instruct","messages":[{"role":"user","content":"Hello"}]}'

If the key is missing or incorrect, the server returns HTTP 401 with a JSON error body:

{"error": {"message": "Invalid API key", "type": "invalid_api_key"}}

The LLAMA_API_KEY environment variable is an alternative to the --api-key flag and is useful in containerized deployments.

Request Body Errors¶

/v1/chat/completions, /v1/responses, /v1/messages, and /v1/messages/count_tokens, together with their aliases without /v1, answer a body they cannot parse the same way /v1/completions does: with a JSON invalid_request_error carrying the parser's message, truncated to 1 KiB. This happens before the availability check, dispatch, or streaming setup, and authentication still runs first. Malformed or empty JSON, and a missing or wrongly typed field such as model, return HTTP 400; a missing or non-JSON Content-Type returns HTTP 415. The Anthropic routes use their own envelope, {"type": "error", "error": {...}}, and /v1/messages/count_tokens answers exactly as /v1/messages does. The disaggregated router's /v1/chat/completions behaves the same way, including HTTP 413 for a body over its size limit. On the single-node server, one server-wide layer answers every HTTP 413 in the OpenAI envelope, on the Anthropic routes too.

Anthropic Messages¶

Claude Code starts its system prompt with an x-anthropic-billing-header stamp whose cch value changes on every request and whose fourth cc_version part changes on every session. On the Anthropic Messages routes the server replaces those two values with f characters of the same length before rendering, so a turn reuses the whole previous prompt instead of stopping at the stamp. Only top-level system text starting with the exact header is touched. Set MLXCEL_ANTHROPIC_PIN_BILLING_STAMP=0 to pass the client's text through unchanged. The variable is read once at startup through a OnceLock, so changing it afterwards has no effect.

Endpoint Overview¶

The server exposes the following HTTP endpoints:

The /slots endpoint is enabled by default and can be disabled with --no-slots. The /props and /metrics endpoints are disabled by default; enable them with --props and --metrics. Prompt-cache endpoints are always mounted and return a stable disabled payload when the cache is off.

For full request and response schemas, see the API Reference and Embedding and Reranking APIs.

Serving Audio Models¶

Whisper and Kokoro checkpoints populate an audio-only server slot. A Whisper process serves transcription and translation; a Kokoro process serves speech synthesis. Neither process also serves chat generation.

# STT
mlxcel-server -m ./models/whisper-base \
  --audio-queue-depth 8 --audio-request-timeout-secs 120

# TTS
mlxcel-server -m ./models/kokoro-82m \
  --audio-queue-depth 8 --audio-request-timeout-secs 120

See OpenAI Audio API for checkpoint layouts, request schemas, supported WAV formats, voice packs, limits, and error responses.

Serving Nemotron VoiceChat (Full-Duplex Speech)¶

A Nemotron VoiceChat checkpoint (model_type: "nemotron_voicechat", for example mlx-community/NemotronLabs-VoiceChat-11B-4bit) is served over a WebSocket at /v1/realtime instead of the chat routes:

mlxcel-server -m models/NemotronLabs-VoiceChat-11B-4bit --port 8080

The server loads the checkpoint on a dedicated realtime engine thread and starts neither the chat worker, the batch scheduler, nor the text warmup. /v1/chat/completions, /v1/completions, /v1/responses, and /v1/messages answer 501 Not Implemented with a message naming /v1/realtime.

• --api-key applies to the upgrade request (send Authorization: Bearer <key>), and --api-prefix prefixes the path like every other route.
• --cors-origins and --allowed-origins gate the upgrade by its Origin header. A browser origin the policy does not allow gets 403 Forbidden; a client that sends no Origin is accepted.
• The engine admits one session at a time. A second connection while a session is active receives a server_busy error and close code 1013.
• Limits: 1 MiB per WebSocket message or frame, 600 s of audio per session, 30 s to send session.update after connecting, and 120 s without a client message before an idle session is closed.

The CLI form of the same model is mlxcel generate --audio <wav> --output-audio <wav> (see CLI Reference). See Nemotron VoiceChat Realtime API for the message protocol.

Serving Embedding and Reranking Models¶

The server can run an embedding model and a reranker beside a chat model. Each specialized provider owns its model, tokenizer, and MLX arrays on a dedicated worker thread, so neither shares the chat worker's stream.

mlxcel-server \
  -m mlx-community/Qwen3-4B-4bit \
  --embedding-model Qwen/Qwen3-Embedding-0.6B \
  --reranker-model mlx-community/Qwen3-Reranker-0.6B-4bit \
  --port 8080

An embedding checkpoint supplied directly to -m starts an embedding-only process; a one-label ForSequenceClassification checkpoint starts a rerank-only process. Generative rerankers require explicit selection because their config resembles a generator:

# Generative reranker only; the duplicate path selects the kind without loading twice.
mlxcel-server \
  -m mlx-community/Qwen3-Reranker-0.6B-4bit \
  --reranker-model mlx-community/Qwen3-Reranker-0.6B-4bit

One process supports one embedding provider and one reranker provider. --embedding-batch-size defaults to 16. --rerank-batch-size 0 selects the kind default of 8 text pairs or 2 multimodal pairs. Each worker queue is bounded by --embedding-queue-depth (default 8) and both request types use --embedding-request-timeout-secs (default 120). A full queue returns 503 and a reply timeout returns 504.

--pooling and --embd-normalize override the checkpoint's dense-vector contract. Use them only when the model's documentation or a compatibility requirement calls for it. --embedding/--embeddings and --rerank/--reranking select restricted llama-server-compatible modes and require the matching provider.

See Embedding and Reranking APIs for exact wire shapes and Embeddings and Reranking for model families, prefix rules, MaxSim, index design, and evaluation.

Serving an Image Model¶

--image-model <checkpoint> (or MLXCEL_IMAGE_MODEL) loads an image-generation model beside the chat model, on its own worker thread with a bounded queue, the same way --embedding-model loads an embedding provider. Requests run one at a time, because a diffusion run holds the GPU for seconds: a full queue answers 503 and a per-request timeout answers 504. A load failure is fatal at startup, so a server that was asked for image routes never answers 501 for them.

mlxcel-server -m mlx-community/Qwen3-4B-4bit --image-model ./models/flux2-klein

Dropping -m selects image-only serving. The image checkpoint becomes the served model, and its id is the first --alias entry or the checkpoint directory name:

mlxcel-server --image-model ./models/flux2-klein --port 8080
mlxcel serve --image-model ./models/flux2-klein

/health reports ready once the checkpoint loads, /v1/models lists that id with the image capability and no completion, and the /v1/images/* routes run normally. Chat, completion, prompt-inspection, /tokenize, /detokenize and /apply-template answer the same structured 501 that embedding-only mode uses, naming --image-model and no -m. Image-only mode cannot be combined with --models-dir, --models-preset, --webui, --embedding-model, --reranker-model, --embeddings or --reranking; startup fails naming both flags.

The routes, checkpoint families, and request schemas are in Image Generation.

Server Lifecycle¶

The server goes through a defined sequence of lifecycle states from startup to shutdown.

sequenceDiagram
    participant CLI
    participant Startup
    participant ModelLoader
    participant BatchScheduler
    participant HTTP

    CLI->>Startup: mlxcel serve --model ...
    Startup->>Startup: Initialize logging
    Startup->>Startup: Resolve API key
    Startup->>Startup: Resolve chat template
    Startup->>ModelLoader: Load model weights
    ModelLoader-->>Startup: Model ready
    Startup->>BatchScheduler: Start batch scheduler
    Note over Startup: Warmup pass (1 token)
    Startup->>HTTP: Bind TCP / Unix socket
    HTTP-->>CLI: Server listening on :8080

    loop For each request
        HTTP->>BatchScheduler: Enqueue request
        BatchScheduler->>ModelLoader: Prefill + decode
        ModelLoader-->>HTTP: Token stream
        HTTP-->>CLI: SSE / JSON response
    end

    CLI->>HTTP: SIGINT / SIGTERM
    HTTP-->>CLI: Graceful shutdown

Lifecycle state descriptions:

1. Logging initialization - Tracing subscriber is configured based on --verbose and --log-file flags.
2. API key resolution - The key is read from --api-key, --api-key-file, or LLAMA_API_KEY.
3. Chat template resolution - The Jinja2 template is loaded from --chat-template, --chat-template-file, or the model's tokenizer_config.json.
4. Model loading - Safetensor weight files are read and their tensors uploaded before the server reports Model loaded, so the first request pays no weight I/O (MLXCEL_WEIGHT_MATERIALIZATION=lazy defers this to first use). The scheduler logs the decode path per model, for example decode path: native batched forward. If --surgery is set, the configured weight surgery pipeline is applied during loading. If --adapter is set, LoRA weights are fused through the adapter path.
5. Batch scheduler start - The continuous batching scheduler is initialized with the configured queue depth and batch size limits.
6. Warmup pass - A single one-token generation pass is run to trigger Metal shader compilation. This prevents the first real request from experiencing compilation latency. Disable with --no-warmup.
7. Socket binding - The server binds to the TCP address or Unix socket and begins accepting connections.

The server does not have an explicit graceful shutdown handler beyond standard signal propagation. Active requests that are in flight when the process receives SIGINT or SIGTERM are interrupted. On exit, the server joins its chat model worker and every side worker (audio, embedding, image, rerank, and realtime) before main returns, because a thread that frees MLX arrays after the C runtime's static destructors run aborts a CPU build with pure virtual method called. The join is bounded by a 120 s WORKER_EXIT_TIMEOUT (150 s for the router's pool drain). A worker still loading when the deadline expires keeps running, and the server logs that it is still running rather than reporting a clean exit.

Monitoring¶

Health Check¶

The /health endpoint returns a JSON body describing the server status. No authentication is required, making it suitable for load balancer health probes. It returns HTTP 200 when the model is loaded and capacity is available, and HTTP 503 while the model is still loading or no slot is available.

curl -s http://localhost:8080/health | jq .

{
  "status": "ok",
  "model": "Qwen2.5-7B-Instruct",
  "batch": {
    "active_sequences": 2,
    "queue_depth": 0,
    "max_batch_size": 4
  },
  "observability": {
    "sequences_started": 145,
    "sequences_completed": 143
  },
  "context_size": 0,
  "tool_call_parser": "mlxcel"
}

context_size is the effective per-slot context window. With an explicit --ctx-size and multiple active slots, it reports the shared budget after division, not the original total budget.

Prometheus Metrics¶

When started with --metrics, the server exposes a /metrics endpoint in Prometheus text format:

mlxcel serve --model ./models/Qwen2.5-7B-Instruct --metrics
curl -s http://localhost:8080/metrics | grep mlxcel

Metrics include batch occupancy, request queue depth, tokens generated, and request completion counts.

mlxcel_requests_total and mlxcel_completion_tokens_total count a chat completion the same way whether it streams or not. A streamed chat completion is recorded once when its generation finishes, with the token counts of its usage chunk, whether or not the request asked for that chunk. A stream the client cancelled is not counted: a disconnect of a stream sent without X-Conversation-Id, DELETE /v1/stream, or replacement of the session by a new request with the same X-Conversation-Id. A resumable stream whose client only disconnects keeps generating and is counted when it finishes. Streaming /v1/completions and native /completion requests are not recorded in these counters.

Slot Status¶

By default, the /slots endpoint shows the current state of each concurrency slot, including the same effective per-slot context_size reported by /health. Disable it with --no-slots if needed:

curl -s http://localhost:8080/slots | jq .

Integration Examples¶

curl¶

Streaming chat completion:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Count from 1 to 5."}],
    "stream": true
  }'

Non-streaming text completion:

curl -s http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen2.5-7B-Instruct",
    "prompt": "The capital of France is",
    "max_tokens": 5
  }'

Python requests¶

import requests

response = requests.post(
    "http://localhost:8080/v1/chat/completions",
    json={
        "model": "Qwen2.5-7B-Instruct",
        "messages": [{"role": "user", "content": "What is Rust?"}],
        "max_tokens": 200,
    },
)
print(response.json()["choices"][0]["message"]["content"])

OpenAI Python SDK¶

mlxcel's /v1/chat/completions endpoint is schema-compatible with the OpenAI API. Point the base_url at the mlxcel server:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="sk-any-value",   # required by the client library; ignored if server has no key set
)

response = client.chat.completions.create(
    model="Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Explain monads in simple terms."}],
    max_tokens=300,
)
print(response.choices[0].message.content)

Streaming with the OpenAI SDK:

stream = client.chat.completions.create(
    model="Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Write a haiku about Rust."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Request Priority¶

The server supports three priority levels for the prefill queue: high, normal (default), and low. Set the X-Priority header to control queue ordering:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Priority: high" \
  -d '{"model":"Qwen2.5-7B-Instruct","messages":[{"role":"user","content":"Urgent query"}]}'

High-priority requests move ahead of normal and low-priority requests in the prefill queue. This is useful when some clients (for example, interactive UI requests) need lower latency than background batch jobs.

Reproducible Greedy Output (cache_prompt)¶

A prompt-cache hit is a split prefill: the cached prefix came from an earlier forward and only the remainder is prefilled now. The split changes the floating-point summation order, so the first generated token's logits are close to a cold prefill's but not always bit-identical, and a near-tie can flip a greedy reply. This is not cache corruption, and no split size removes it.

When a request needs output that does not depend on cache state (regression tests, evaluation runs, comparing two builds), send cache_prompt: false. The request is prefilled cold and neither reads nor donates cache entries. The field is honored at the request root, through the flattened OpenAI-SDK extra_body, or nested inside extra_body, on the chat routes and on native /completion. --no-cache-prompt is the server-wide form.

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"m","temperature":0,"cache_prompt":false,"messages":[{"role":"user","content":"Hello"}]}'

See Prompt Cache for when a cache hit reproduces the cold run exactly.

Long Prefill and Proxy Timeouts¶

When a prompt is large (32k+ tokens), the prefill forward pass can take tens of seconds before the first generated token is emitted. During that window the SSE stream is open but silent.

Reverse proxies and load balancers apply per-connection idle timeouts that fire before the first token arrives:

mlxcel-server sends an SSE keepalive comment every 15 seconds to prevent these drops. The comment is invisible to clients that parse SSE token events; it only resets the proxy idle timer.

If you see clients receiving 504 or connection-reset errors on long-context requests, the first thing to check is whether the proxy idle timeout is shorter than the prefill time. Raising proxy_read_timeout to 300+ seconds (or disabling the timeout on the tunnel leg) is the most reliable fix for very long prompts. The --decode-timeout flag controls the server-side decode-hang detection (default: 600 s), not the prefill step; setting it does not prevent proxy drops. --timeout is the HTTP socket read/write timeout (default: 3600 s).

KV Cache Size Limit (--max-kv-size)¶

By default, the KV cache grows without bound for the lifetime of each sequence. On long-context conversations or generation tasks that exceed the model's context window, memory footprint grows proportionally with the number of tokens processed. --max-kv-size N caps the live KV window at N tokens: once a sequence's cache exceeds N tokens the scheduler drops the oldest entries before each forward pass so only the most recent N tokens participate in attention.

RoPE-faithful semantics: the cache's monotonic write position (offset) is never decremented. When the live window is trimmed, live_start advances instead, so every cached key's relative position to the current query stays correct under RoPE. This matches mlx-lm's position semantics and avoids the position drift that a naive offset-decrement approach would introduce.

CLI flag (applies server-wide to all sequences):

mlxcel-server --model ./models/Llama-3.1-8B-Instruct --max-kv-size 4096

Environment variable (CLI flag wins on conflict):

LLAMA_ARG_MAX_KV_SIZE=4096 mlxcel-server --model ./models/Llama-3.1-8B-Instruct

Valid range: 0 (disabled, the default) or any integer from 64 to 2147483647 (i32::MAX). Values below 64 are rejected at startup because they are smaller than the prefill step granularity and would cause the cache to trim on every decode step without providing a useful live window. Values above i32::MAX are rejected because the MLX FFI uses i32 for all cache shape and slice indices.

Sliding-window models are unaffected: models that already manage their own rotating window (RotatingKVCache), namely Gemma ¾, Exaone 4, RecurrentGemma, Step 3.5, and gpt-oss, apply their own model-level cap and ignore --max-kv-size.

Turbo KV quantization limitation: --max-kv-size is not supported together with Turbo KV quantization (--kv-cache-mode turbo4* / --kv-quant-scheme turboquant). When both are set, the live-window cap is silently skipped for Turbo-quantized layers and a warning is logged at startup. Interaction with the paged memory backend is out of scope for this release and tracked separately.

Nemotron-H Request Memory Admission¶

On Metal, text-only Nemotron-H serving checks a conservative memory estimate before it starts each request. The scheduler clamps this family to one active slot. A request whose estimate exceeds the budget never runs GPU forward work and gets HTTP 503 with error type request_memory_budget_exceeded; a streaming request receives the 503 before SSE headers are sent. /health stays available, and smaller requests run afterwards. Existing context-length checks take precedence and stay HTTP 400.

The budget is the smallest of the device's recommended working-set size, the configured wired-memory cap, and the current MLX allocator cap. MLXCEL_REQUEST_MEMORY_BUDGET (for example 24G) can lower it but never raise it; an explicit zero, empty, none, negative, nonfinite, or overflowing value stops server startup. The estimate covers resident weights and retained prompt snapshots, cold KV for prompt plus planned output, recurrent and convolution state, and the largest forward of the request's prefill plan.

Guarded requests prefill cold: they neither restore nor donate prompt snapshots, and background prompt-cache warmups are skipped. Both the batch scheduler and --no-batch serving apply the guard. Metal Nemotron-H runtime (unfused) LoRA fails at model construction because its activation workspace is not modeled; use a fused adapter checkpoint instead. The guard does not apply to CUDA, ROCm, the CPU backend, other model families, multimodal paths, or speculative decoding, and it is not a device-wide reservation against memory used by other processes.

Loop Detection (N-gram Repetition Guard)¶

Some checkpoints can collapse into a degenerate loop under tool- or grammar-constrained decoding, where a single token or a short block repeats until the token budget is exhausted. mlxcel can detect a short pattern repeating at the tail of the generated stream and end the request early, the same countermeasure vLLM exposes in SamplingParams.

The Gemma 4 family enables this unconditionally by default (threshold min=1, max=20, count=4), independent of whether the request uses tools or a json_schema response format. Other families leave it off unless a request opts in.

Control it at three levels, each overriding the one before it:

1. Per-family default: on for Gemma 4, off otherwise.
2. Operator override (MLXCEL_LOOP_DETECTION): forces the policy for every request on this server. See MLXCEL Environment Variables.
3. Per-request fields (max_pattern_size, min_pattern_size, min_count in the request body): always win, including an explicit max_pattern_size: 0 to force-disable on a Gemma 4 server. See REST API Reference.

Reasoning Re-echo for Content-Only Clients¶

Some chat templates render an earlier turn differently depending on whether the following assistant message carries its reasoning. AI21 Jamba-Reasoning, for example, keeps its thinking instruction on an earlier user turn only in that case, so a client that sends back only content (the OpenAI SDK default) re-renders that turn shorter than it was generated and the follow-up misses the prompt cache.

While the prompt cache is on, the chat routes remember the exact reasoning they returned for each plain (non-tool-call) reply and put it back into a later content-only assistant turn, as if the client had echoed it:

A stored trace is found only by a request with the same model, chat-template signature (template, kwargs, tools), session (prompt_cache_key, then user), identical preceding messages, and identical assistant content. Any edit misses, a miss changes nothing, and reasoning the client echoes itself always wins. Echoed reasoning reaches the chat template under both reasoning_content and reasoning, so templates that read either field render it once. The store holds only a 32-byte key and the reasoning text, is bounded to 16 MiB and 4096 entries with least-recently-used eviction (MLXCEL_REASONING_ECHO_MAX_BYTES, 0 disables it), and is empty after a restart. It is not consulted under cache_prompt: false, --reasoning-format none or deepseek-legacy, or --skip-chat-parsing.

Templates whose generation prompt ends in <think>\n (primed-think templates) stream the model's reasoning as reasoning from the first token on all three routes.

Thinking-Token Budget (Qwen3 family)¶

Qwen3, Qwen3.5, and Qwen3.6 generate a <think>...</think> reasoning block before the final answer. mlxcel-server caps how many tokens the model can spend inside that block, matching the semantics of llama.cpp's --reasoning-budget flag and vLLM's thinking_token_budget sampling parameter.

The opening <think> token is not counted toward the cap (matching vLLM). For models without <think> / </think> tokens the parameter is silently ignored, and the server logs one warning per model load.

CLI flag (applies to every request unless overridden per-request):

mlxcel-server --model ./models/Qwen3-8B-Instruct --reasoning-budget 256

Environment variable (CLI flag wins on conflict):

LLAMA_ARG_REASONING_BUDGET=256 mlxcel-server --model ./models/Qwen3-8B-Instruct

Per-request override on /v1/chat/completions. The primary name is thinking_budget_tokens; thinking_token_budget (vLLM) and thinking_budget (Qwen) are accepted as aliases. When more than one is present, the primary name wins:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "qwen3",
        "messages": [{"role":"user","content":"Explain gradient descent."}],
        "max_tokens": 512,
        "thinking_budget_tokens": 256
      }'

Per-request override on /completion (llama.cpp-style). The same three alias names are accepted:

curl http://localhost:8080/completion \
  -H "Content-Type: application/json" \
  -d '{
        "prompt": "<|im_start|>user\nExplain gradient descent.<|im_end|>\n<|im_start|>assistant\n",
        "n_predict": 512,
        "thinking_budget_tokens": 256
      }'

Resolution precedence (highest wins):

1. Per-request body field (thinking_budget_tokens, then thinking_token_budget, then thinking_budget).
2. --reasoning-budget <N> CLI flag.
3. LLAMA_ARG_REASONING_BUDGET env var.
4. Default: unrestricted (-1).

A per-request value of -1 reverts to unrestricted for that one request, even if the server default was set. When thinking_budget_tokens > 0 it must be <= max_tokens (or n_predict on /completion); otherwise the server returns a 400. Other negative values (e.g. -2) also return 400.

When the budget fires, the forced </think> is emitted as a normal SSE delta chunk in streaming mode and counted toward finish_reason accounting (it never triggers user-supplied stop sequences).

Chat Template Kwargs¶

Some HuggingFace chat templates accept feature switches rendered directly into the prompt. mlxcel-server accepts a JSON object of template kwargs at startup (as a default) and per-request (as an override), matching the shape of llama.cpp's --chat-template-kwargs flag and vLLM's extra_body.chat_template_kwargs. Today the canonical consumer is Qwen3.6's preserve_thinking for multi-turn reasoning retention, but the mechanism is generic so future keys inherit the same plumbing.

CLI flag:

mlxcel-server --model ./models/Qwen3.6-35B-A3B \
  --chat-template-kwargs '{"preserve_thinking": true}'

Environment variable (CLI wins on conflict):

LLAMA_ARG_CHAT_TEMPLATE_KWARGS='{"preserve_thinking": true}' \
  mlxcel-server --model ./models/Qwen3.6-35B-A3B

Per-request override on /v1/chat/completions. Three shapes are accepted; the first match wins:

1. Top-level chat_template_kwargs (llama.cpp shape):

{"chat_template_kwargs": {"preserve_thinking": true}}

1. Nested under extra_body (vLLM / OpenAI Python SDK shape):

{"extra_body": {"chat_template_kwargs": {"preserve_thinking": true}}}

client.chat.completions.create(
    model="qwen3.6",
    messages=[...],
    extra_body={"chat_template_kwargs": {"preserve_thinking": True}},
)

1. DashScope flat shape (secondary; only recognized for the preserve_thinking key):

{"extra_body": {"preserve_thinking": true}}

Merge semantics: per-request keys override the server default on a per-key basis; unrelated server-default keys persist through the merge. If the server is started with {"preserve_thinking": true, "enable_thinking": false} and a request supplies {"preserve_thinking": false}, the effective kwargs are {"preserve_thinking": false, "enable_thinking": false}.

Resolution precedence (highest wins):

1. Top-level chat_template_kwargs on the request.
2. extra_body.chat_template_kwargs on the request.
3. extra_body.preserve_thinking on the request (DashScope, preserve_thinking key only).
4. --chat-template-kwargs CLI flag.
5. LLAMA_ARG_CHAT_TEMPLATE_KWARGS env var.
6. Default: empty (no kwargs).

preserve_thinking behavior:

• true: <think>...</think> blocks from every prior assistant turn reach the model unchanged.
• false (default): assistant messages before the latest non-tool user turn have their <think> blocks stripped server-side, matching the Qwen3/Qwen3.5 rolling-checkpoint convention.

Non-thinking models (Qwen2, Llama) silently ignore the flag. Qwen3 / Qwen3.5 accept the flag, but the multi-turn quality benefit is only validated on Qwen3.6 because earlier Qwen3 variants were not trained on preserved reasoning traces.

Malformed JSON on --chat-template-kwargs or LLAMA_ARG_CHAT_TEMPLATE_KWARGS causes the server to refuse to start with a clear error.

enable_thinking auto-default for thinking models:

At startup, after loading the tokenizer, the server calls infer_thinking_markers() to resolve the model's think-start / think-end token sequences. If a matching pair is found, the server automatically seeds enable_thinking=true as a server-level default kwarg. Recognized pairs are single-token <think> / </think> (Qwen3, GLM4, Nemotron-H, ExaOne4), single-token <longcat_think> / </longcat_think> (some fine-tunes), and multi-token <|channel>thought / <channel|> (Gemma 4). That default flows through build_template_context the same way any other server-default kwarg does, so per-request chat_template_kwargs or extra_body.chat_template_kwargs can still override it.

For non-thinking models the flag is never injected, leaving their chat templates unaffected. The server emits one INFO log line at startup identifying the resolved marker pair and the number of tokens in each sequence, which is useful for verifying that a Gemma 4 multi-token marker is being resolved correctly:

INFO Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` think_start=Some("<|channel>thought") think_end=Some("<channel|>") think_start_tokens_len=2 think_end_tokens_len=1

To disable thinking mode for an individual request on a thinking model, pass enable_thinking=false in any of the three per-request shapes described above.

reasoning_effort mapping: the chat-completions reasoning_effort field is mapped onto the reasoning_effort chat-template kwarg under three guards. An explicit chat_template_kwargs.reasoning_effort wins, the loaded template must actually mention the name, and the value is passed through rather than translated. /v1/responses does not do this mapping for its reasoning.effort field; set chat_template_kwargs explicitly there.

A template that refuses a request fails the request. When rendering the chat template raises, the request returns HTTP 400 before generation starts rather than falling back to a plain concatenation that the model was not trained on. This covers the typed path and the raw/multimodal path alike, so a request carrying tool calls, a prior-turn reasoning field, or typed media parts gets the same answer. A streaming request returns the 400 in place of the SSE stream rather than failing mid-stream.

Serving Florence-2¶

Florence-2 is an encoder-decoder model, so the server routes it to a dedicated single-stream seq2seq worker rather than the batch or paged scheduler. Requests queue on the worker channel, bounded by --max-queue-depth, and are answered one at a time by design: the encoder pass has a different cost profile from the decode loop, and no batched admission policy has been measured for it.

Send an OpenAI-compatible POST /v1/chat/completions with one user message whose text is the Florence-2 task prompt and one image_url content part. message.content carries the rendered answer and the mlxcel-specific message.florence2_result field carries the parsed coordinates as JSON.

Decode is greedy, so sampling parameters and response_format are accepted and ignored, and stream: true delivers the whole answer as a single delta chunk because the parsed result only exists after the full decode. The input text of the seven input-taking task modes is validated at the request boundary, all declared images must resolve, and image payloads pass through the same decompression-bomb bounds every other family uses. usage.prompt_tokens reports the fused encoder length including the projected image tokens.

See Vision-Language Models for the task markers and the CLI form.

Serving Block-Diffusion Models¶

When the loaded checkpoint is a block-diffusion model (DiffusionGemma), the server routes requests to a dedicated serial worker instead of the batch scheduler: requests are queued in arrival order and processed one at a time, by design. /v1/chat/completions and /v1/completions both work, with streaming and non-streaming responses, and image input follows the standard OpenAI image_url content part format.

Three serve-level flags tune the diffusion engine: --diffusion-sampler, --diffusion-threshold, and --max-denoising-steps. The checkpoint is a thinking-channel model, so give requests enough max_tokens (a budget of 64 may produce only reasoning with an empty content; 256 covers most short prompts). See Block Diffusion for the generation mechanism, flags, and operational caveats.