CLI Reference¶

mlxcel exposes nineteen subcommands. run and generate produce model output; generate-image renders images; embed creates dense or multi-vector representations; rerank scores query-document pairs; detect finds objects; predict writes dense maps and keypoints; tag and extract run fixed-label token classification and request-time span extraction; inspect sizes memory before loading; serve starts the HTTP server; list, arch, download, and rm manage or inspect model availability; tune profiles and caches kernel launch configurations; split-mtp writes a standalone MTP drafter out of a raw GLM-4.7-Flash checkpoint; and prepare-external-ple and prepare-expert-stream build the on-disk structures that qwen4_exp reads outside its MLX parameters. generate has the gen alias; list has the ls alias; arch has the supported alias.

Table of Contents¶

• Environment Variables
• mlxcel run
• mlxcel generate
• Model Options
• Generation Options
• KV Cache Options
• Sampling Options
• Prompt Lookup Options
• Diffusion Options
• Pipeline Parallel Options
• Tensor Parallel Options
• Model Surgery Options
• Language Bias Options
• Examples
• mlxcel inspect
• mlxcel detect
• mlxcel predict
• mlxcel generate-image
• mlxcel embed
• mlxcel rerank
• mlxcel tag
• mlxcel extract
• mlxcel serve
• Network and Auth
• Concurrency and Batching
• Embedding and Reranking Workers
• Speculative Decoding
• Endpoint Toggles
• Model Resolution and Store
• Router Mode
• WebUI
• LoRA Adapters
• Structured Output Defaults
• Default Sampling Parameters
• Distributed Inference
• Examples
• mlxcel list
• mlxcel arch
• mlxcel download
• mlxcel rm
• mlxcel tune
• mlxcel split-mtp
• mlxcel prepare-external-ple
• mlxcel prepare-expert-stream

Environment Variables¶

These variables are read at startup before any flags are processed.

Setting MLXCEL_WIRED_LIMIT=max can improve throughput for large models but risks out-of-memory conditions if the model and working tensors together exceed physical memory.

mlxcel run¶

mlxcel run [OPTIONS] [MODEL_OR_REPO_ID]

Runs a model with an ollama-style positional model argument. Without -p/--prompt, it opens the interactive multi-turn chat REPL. With -p, it performs one generation and exits. When the model argument is omitted, it uses the default mlx-community/gemma-4-e2b-it-4bit.

MODEL_OR_REPO_ID accepts the same resolver inputs as generate -m: an existing local path, a HuggingFace owner/name repo-id, or a bare name resolved as ${MLXCEL_DEFAULT_ORG:-mlx-community}/<name>. Repo-id misses are downloaded into the mlxcel model store and reused from any directory.

Examples:

# Interactive chat, auto-downloading on first use.
mlxcel run Qwen3.5-0.8B-4bit

# One-shot generation.
mlxcel run Qwen3.5-0.8B-4bit -p "Hello" -n 100

# Override the default owner used by bare names.
MLXCEL_DEFAULT_ORG=my-org mlxcel run My-Model-4bit

How run executes¶

run and its REPL are clients of the mlxcel-server model worker, started inside the same process with one slot (--parallel 1, --max-batch-size 1) and no HTTP listener. Each turn goes through the /v1/chat/completions request path, which renders the chat template, splits reasoning from content, parses tool calls, and applies stop strings. For the same request, mlxcel run -p and mlxcel-server produce the same tokens; under MLXCEL_SDPA_DETERMINISTIC=1, run -p --temp 0 and the server give identical greedy output. See Token-exactness between the CLI and the server.

run -p prints only the reply (not the prompt), followed by a [Generated N tokens ...] summary line and, when a drafter is active, a [Speculative ...] acceptance line. The prompt cache is off for run -p and on for the REPL, where each conversation carries its own prompt_cache_key so a follow-up turn prefills only the new messages.

Checkpoints that ship no chat template are rendered with the server's generic chat format, and run prints a notice that the checkpoint is likely a base model. --no-chat-template sends the raw text through the /v1/completions path instead.

A few one-shot modes that the chat server does not serve run on the generate flow instead: block-diffusion, Florence-2, Nemotron-Parse, and Nemotron VoiceChat checkpoints, and the --output-audio, --layout-detections, --audio, --video, --profile, --estimate-memory, and --recommend-quant flags. --no-chat-template combined with --image also stays on generate. --image otherwise goes through the server like a chat client's image_url part.

Flag groups on run¶

run accepts the generate generation, media, KV cache, diffusion, and sampling options, plus the server sampling options below and the speculative flags in the table above. It does not take the pipeline-parallel, tensor-parallel, language-bias, model-surgery, or prompt-lookup groups. Use mlxcel run --help as the exhaustive flag list.

Each sampling flag maps onto the mlxcel-server option of the same name. A flag left unset keeps the server default, with one exception: the base sampler stays greedy unless the checkpoint's generation_config.json sets a temperature. --repeat-last-n and --dry-penalty-last-n are forwarded only when given, so an unset run uses the server's 64-token windows (inert unless a penalty is active), while generate uses -1 (full history).

Interactive commands¶

The whole transcript, images included, is sent again on every turn. /image and --image therefore refuse an image that would push the conversation over the server's per-request image cap (16 by default); /clear resets the count. In raw mode (--no-chat-template) turns carry no media: the REPL refuses /image and ignores --image with a notice.

mlxcel generate¶

mlxcel generate [OPTIONS] --model <PATH_OR_REPO_ID> [--prompt <TEXT>]
mlxcel gen [OPTIONS] --model <PATH_OR_REPO_ID> [--prompt <TEXT>]   # alias

Loads a model, applies a prompt when provided, streams generated tokens to stdout, and prints a summary line. When --prompt is omitted, generate drops into the same interactive multi-turn REPL used by mlxcel run.

generate -p is the raw-completion client. It decodes on the engine's direct client (mlxcel_core::engine::DirectEngine) with the same sampler, finish step, and B=1 lookahead pipeline as the server's single-sequence row; MLXCEL_FORCE_SYNC disables that pipeline. Chat prompts are rendered from the checkpoint's template the way the server renders them. Under --profile, the [TTFT] line reports sample=0 because the sampler build is counted inside eval.

Model Options¶

Generation Options¶

KV Cache Options¶

See TurboQuant KV Cache for compatibility and quality guidance.

Sampling Options¶

These flags control how the next token is chosen from the model's probability distribution.

Prompt Lookup Options¶

Prompt lookup is drafter-free n-gram speculation: each round proposes the tokens that followed an earlier occurrence of the sequence's last few tokens and verifies them in one target forward. It pays off on replies that copy the prompt (edits, rewrites, reformatting); on other replies the proposals are shortened or paused so decoding stays near plain speed. Acceptance is sampler-match, which is lossless for a one-hot proposal; greedy output can differ from plain decoding only at logit near-ties. See Speculative Decoding for the server form (--draft-kind prompt-lookup).

The tuning flags require --prompt-lookup. mlxcel generate --draft-kind prompt-lookup is a parse error that points to --prompt-lookup. A --prompt-lookup run is refused for interactive chat (no -p), image, audio, or video input, pipeline parallelism, and target models whose state a cache trim cannot roll back: recurrent and hybrid models, and models that own their attention caches (Qwen 3.5, Gemma 3 and 4, Llama 4). The first run warms every verify width before the timed generation. On a GB10, edit-style replies measured 1.10x to 1.94x faster than plain decoding, while free prose stays at about 0.98x to 1.00x with the gated policy.

Diffusion Options¶

These flags control block-diffusion generation and only affect diffusion models such as DiffusionGemma; autoregressive models ignore them. See Block Diffusion for the generation mechanism and tuning guidance.

Pipeline Parallel Options¶

These flags enable the in-process pipeline path for mlxcel generate.

Current CLI pipeline notes:

• The path is available to the stage-executor families listed in Pipeline Parallelism, not only Llama.
• It is text-only and does not support speculative decoding or VLM inputs. One load-time adapter can be fused stage-locally on the current Llama stage executor; other families fail explicitly.
• In-process CLI PP×TP composition is supported for eligible dense runtimes, but TP×LoRA is not. Remote pipeline serving has stricter limits and currently rejects adapters and TP composition.

Tensor Parallel Options¶

These flags enable the current in-process tensor-parallel runtime for supported dense text model families.

Supported tp_size > 1 families are dense Llama, Qwen2/2.5, Qwen3, Qwen3.5 text, Gemma 3 text, Gemma 4 text, ERNIE 4.5, and Hunyuan v1 Dense. LoRA adapters are not supported with TP.

Model Surgery Options¶

Runtime model surgery applies a YAML-defined weight transform while the checkpoint is loaded. Use it when you need temporary, reproducible weight adjustments without editing the model files on disk. See Runtime Model Surgery for the operation schema and limitations.

Language Bias Options¶

mlxcel generate and both server entry points expose language-bias controls for suppressing or encouraging scripts at sampling time. See Language Bias for operational guidance.

Generate Examples¶

Basic text generation with greedy decoding:

# Bare name resolves to mlx-community/Qwen2.5-7B-Instruct-4bit.
mlxcel generate \
  --model Qwen2.5-7B-Instruct-4bit \
  --prompt "Explain the difference between a mutex and a semaphore." \
  --max-tokens 300

Interactive chat from generate:

mlxcel generate --model Qwen3.5-0.8B-4bit

Gemma 4 with a larger vision-token budget:

mlxcel generate -m ./models/gemma-4-e2b-it-4bit \
  --image page.png --image-soft-tokens 560 \
  -p "Read the small text in this page."

Falcon-OCR region-wise OCR:

mlxcel detect -m ./models/docling-layout -i page.png --format json > regions.json
mlxcel generate -m ./models/falcon-ocr --image page.png --layout-detections regions.json

Qwen3-Omni speech output:

mlxcel generate -m ./models/qwen3-omni \
  -p "Give a short welcome." --output-audio welcome.wav --speaker chelsie

Creative writing with higher temperature and nucleus sampling:

mlxcel generate \
  --model ./models/Llama-3.1-8B-Instruct \
  --prompt "Write the opening paragraph of a science fiction novel." \
  --max-tokens 200 \
  --temp 0.8 \
  --top-p 0.95

Using a LoRA adapter:

mlxcel generate \
  --model ./models/Mistral-7B-v0.3 \
  --adapter ./adapters/mistral-sql-lora \
  --prompt "Write a SQL query to find the top 10 customers by revenue." \
  --max-tokens 150

Applying runtime model surgery:

mlxcel generate \
  --model ./models/Qwen2.5-7B-Instruct \
  --surgery ./surgery.yaml \
  --prompt "Summarize the deployment checklist." \
  --max-tokens 150

Vision-language model with an image:

mlxcel generate \
  --model ./models/Qwen2.5-VL-7B-Instruct \
  --image ./photos/diagram.png \
  --prompt "Describe what you see in this image." \
  --max-tokens 200

Gemma 4 video input:

mlxcel generate \
  --model ./models/gemma-4-vl \
  --video ./demo.mp4 \
  --fps 2.0 \
  --prompt "Summarize the visible events in this clip." \
  --max-tokens 200

Video decoding uses ffmpeg; install it and ensure it is on PATH before using --video.

Multiple images are supported by repeating the --image flag:

mlxcel generate \
  --model ./models/Qwen2.5-VL-7B-Instruct \
  --image ./before.png --image ./after.png \
  --prompt "What changed between these two images?" \
  --max-tokens 200

Full-model speculation (an ordinary smaller checkpoint as the drafter):

mlxcel generate \
  --model ./models/Llama-3.1-70B-4bit \
  --draft-model ./models/Llama-3.2-1B \
  --draft-kind full-model \
  --prompt "Summarize the key ideas in the Rust ownership model." \
  --max-tokens 400

--draft-kind full-model runs a smaller checkpoint that shares the target's tokenizer from its own KV caches, and the engine's token-only verify loop decides every position. Any readable model directory that is neither MTP nor DFlash is auto-detected as full-model. --draft-kind dflash runs offline only for a Laguna target paired with its Laguna DFlash drafter; the Qwen 3.5, LFM2 DSpark, and Muse Glimmer DFlash pairings run in server mode. --draft-kind mtp is supported for qualified Gemma 4/Unified, Qwen 3.5/Qwen3.8-family, and GLM-4.7-Flash text-only pairs.

Prompt-lookup speculation for an edit-style request (no drafter):

mlxcel generate \
  --model mlx-community/Qwen3-1.7B-4bit \
  --prompt-lookup \
  --prompt "Fix the typos in this paragraph and return it unchanged otherwise: ..." \
  --max-tokens 400

Nemotron VoiceChat offline turn (speech in, speech out):

mlxcel generate -m models/NemotronLabs-VoiceChat-11B-4bit \
  --audio question.wav --output-audio response.wav \
  -p "Be concise and answer in one sentence."

This prints [user] <transcript>, the assistant text, and [function] <text> when the function channel produced any, and writes response.wav (22050 Hz mono PCM16). -n is ignored because the output length equals the input timeline, --seed (default 0 for this family) makes the speech reproducible, and --image and --video are rejected. Add --stream --profile to run the online session and print per-stage latency and the real-time factor. Interactive generate without --audio refuses the checkpoint. See Nemotron VoiceChat Realtime API for the server form.

Memory preflight before loading a model:

mlxcel generate \
  --model ./models/Qwen2.5-7B-Instruct \
  --prompt "Summarize the deployment plan." \
  --max-tokens 1024 \
  --estimate-memory

The preflight uses the rendered prompt token count plus --max-tokens, then compares exact weights, KV cache, and runtime headroom against available memory. See Memory Estimation for details and override guidance.

Profiling mode to measure performance:

mlxcel generate \
  --model ./models/Gemma-3-4B-Instruct \
  --prompt "What is photosynthesis?" \
  --max-tokens 200 \
  --profile

Output with --profile includes a breakdown of timing steps:

[Profile Results]
  Prompt tokens:    12
  Generated tokens: 148
  Prefill:          84.3 ms (142.3 tok/s)
  Decode:           3241.7 ms (45.6 tok/s)

Without --profile, a brief summary is printed:

[Generated 148 tokens in 3.24s = 45.6 tok/s]

Disabling the chat template for raw prompt injection:

mlxcel generate \
  --model ./models/Qwen2.5-7B-Instruct \
  --prompt "<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n" \
  --no-chat-template \
  --max-tokens 50

Single-machine pipeline generate on a supported text model:

mlxcel generate \
  --model ./models/llama-3.2-1b-4bit \
  --prompt "Hello" \
  --max-tokens 64 \
  --temp 0 \
  --no-chat-template \
  --pp-size 2

mlxcel inspect¶

mlxcel inspect --model <PATH_OR_REPO_ID> [OPTIONS]

Prints a pre-load memory breakdown for a model and exits without loading it. The output includes weights, KV cache, runtime headroom, total bytes, available bytes, and whether the configuration is expected to fit. This is the same estimator used by --estimate-memory and --recommend-quant; see Memory Estimation.

Examples:

mlxcel inspect -m ./models/Qwen2.5-7B-Instruct-4bit
mlxcel inspect -m ./models/Qwen2.5-7B-Instruct-4bit --max-tokens 32768
mlxcel inspect -m ./models/Qwen2.5-7B-Instruct-4bit --batch 4 --kv-cache-mode int8
mlxcel inspect --json -m mlx-community/Qwen3-4B-4bit | python3 -m json.tool

The JSON form is intended for recipe builders and schedulers that need hardware-fit inputs. It reports mlxcel_version, model, model_type, family, inputs, weights_bytes, weights_source, kv_bytes_per_token, kv_bytes_total, kv_detail, per_slot_overhead_bytes, activation_bytes, backend_inflight_bytes (non-zero only on ROCm builds, where it is part of headroom_bytes), headroom_bytes, headroom_factor, budget_bytes, total_bytes, and fits. Values the estimator cannot supply are null, not guessed.

mlxcel detect¶

mlxcel detect --model <PATH> --image <PATH> [OPTIONS]

Runs object detection on an image and prints bounding boxes in original-image pixel coordinates. Detection models output boxes (l, t, r, b) with a class label and confidence rather than a token stream, so this is a separate command from generate/serve.

The command loads RT-DETRv2, PP-DocLayoutV3, YOLO11, or RF-DETR checkpoints from a directory containing config.json and model.safetensors. Pre-converted MLX checkpoints (e.g. mlx-community/docling-layout-heron-mlx-bf16) and raw HuggingFace checkpoints are accepted; HF weights are translated to MLX layout at load time.

Class names are resolved from the checkpoint's config.json::id2label, or from config.json's class list for YOLO11; when absent, the numeric class id is shown. PP-DocLayoutV3 detections also carry the model's global reading order.

Examples:

# Document-layout detection on a page image (17-class docling model).
mlxcel detect -m ./models/docling-layout-heron-mlx-bf16 -i page.png

# Higher threshold, JSON output for downstream tooling.
mlxcel detect -m ./models/rt-detr-v2 -i photo.jpg --threshold 0.5 --format json

# OmniParser v2 icon detection with a longest-side resize.
mlxcel detect -m axiom-of-choice/OmniParser-v2-icon-detect -i ui.png --size 1280 --format json

JSON output schema:

{
  "image": "page.png",
  "threshold": 0.3,
  "detections": [
    { "label": "section_header", "label_id": 7, "confidence": 0.89,
      "box": { "l": 390.3, "t": 21.7, "r": 738.5, "b": 41.4 } }
  ]
}

mlxcel predict¶

mlxcel predict --model <PATH> (--image <PATH> | --video <PATH>) --output <DIR> [OPTIONS]

Writes dense task outputs for checkpoints that return more than boxes: depth, surface normals, segmentation, point maps, keypoints, per-instance masks, meshes, and Gaussian splats. Every run writes result.safetensors and meta.json into the output directory, plus task-specific files such as keypoints.json, depth.png, normal.png, segmentation.png, instances.json, and masks/.

# Image-encoder features from DINOv2.
mlxcel predict -m ./models/dinov2-small -i input.png -o features --task features

# Pose from Sapiens2 with an explicit box.
mlxcel predict -m ./models/sapiens2-pose -i person.png -o pose \
  --opts '{"boxes":[[140,90,920,700]],"flip_test":true}'

# Depth over a video.
mlxcel predict -m ./models/video-depth -v clip.mp4 -o depth --max-frames 32

Exactly one of --image and --video is required, and --output is required. --task may be omitted when the selected family declares a default for that input. --prompt repeats, --boxes accepts pixel x1,y1,x2,y2 boxes (repeat the flag or separate boxes with ;), --mask and --pointmap supply SAM 3D Objects inputs, --max-frames and --target-fps control video sampling, and --opts carries model-specific options as a JSON object.

The supported families, their tasks, and the output contract are in Dense Prediction and Vision Tasks.

mlxcel generate-image¶

mlxcel generate-image -m <PATH> -p <TEXT> [--task generate|edit] [OPTIONS]

Generates an image from a prompt, or edits one or more reference images, with a diffusion checkpoint in Diffusers layout. The seed is always printed; with -n greater than one, image i uses seed seed + i (except FLUX.2 generation, which draws one noise batch) and is written as <stem>-01.png, <stem>-02.png, and so on.

mlxcel generate-image -m ./models/flux2-klein -p "a watercolor fox"
mlxcel generate-image -m ./models/flux2-klein -p "a fox" --size 768x512 --seed 7 -n 2 -o fox.png
mlxcel generate-image -m ./models/flux2-klein --task edit --image fox.png -p "make it night"

Key flags: --task generate|edit (default generate), --image (repeatable, required for edit and rejected for generate), --size WxH (default 512x512), --negative-prompt, --steps, --guidance, --seed, -n (1 to 10), -o/--output (default outputs/image-<seed>.png), and --gen-kwargs for model-specific options as JSON.

The families, their defaults, and the /v1/images routes are in Image Generation.

mlxcel tag¶

mlxcel tag -m <PATH_OR_REPO_ID> [OPTIONS] <TEXT>

Runs a fixed-label BIO or BIOES token classifier over the text and prints the tagged spans with Unicode character offsets plus a redacted copy. - as the text argument reads it from stdin. --argmax forces per-token decoding on a checkpoint that defaults to a constrained Viterbi decoder, --replacement sets the redaction string, --keep lists labels that stay in the redacted text, and --operating-point selects a calibration point on a Viterbi-decoded checkpoint.

mlxcel tag -m nativ-community/rampart-mlx-fp16 --keep CITY,STATE,ZIP_CODE \
  "I'm Sarah Connor, 1984 Cyberdyne Ave, Los Angeles CA 90012, sarah@sky.net"

mlxcel extract¶

mlxcel extract -m <PATH_OR_REPO_ID> (--labels <L1,L2> | --labels-json <JSON> | --classify <JSON>) <TEXT>

Extracts spans for labels chosen per request from a span-extraction checkpoint, or classifies the text. --include-confidence and --include-spans add each span's score and offsets; --threshold (default 0.5) sets the minimum entity score and the default multi-label threshold; --max-len and --word-splitter whitespace|char apply to GLiNER2.5.

mlxcel extract -m fastino/gliner2.5-small-v1 --labels company,person,location \
  --include-confidence --include-spans "Apple hired Sam in New York."

Both commands, their supported families, and their output shapes are in Token Classification and Span Extraction.

mlxcel embed¶

mlxcel embed -m <MODEL_OR_REPO_ID> -p <TEXT> [-p <TEXT> ...] [OPTIONS]

Loads an embedding checkpoint and prints one vector per input. With two or more inputs, it also prints a cosine-similarity matrix for dense vectors or a MaxSim matrix for late-interaction vectors. The command uses the same detector, loader, pooling, normalization, length limits, and micro-batching as the server.

mlxcel embed -m nomic-ai/modernbert-embed-base \
  -p "search_query: What is vector search?" \
  -p "search_document: Vector search retrieves nearby embeddings."

mlxcel rerank¶

mlxcel rerank -m <MODEL_OR_REPO_ID> -q <QUERY> -d <DOCUMENT> [-d <DOCUMENT> ...] [OPTIONS]

Scores every document against one query, then prints request-order scores and the final ranking. It supports cross-encoder, Qwen3 generative, and Qwen3-VL multimodal rerankers.

mlxcel rerank -m BAAI/bge-reranker-v2-m3 \
  -q "What is vector search?" \
  -d "Vector search retrieves nearby embeddings." \
  -d "A compiler translates source code." \
  --top-n 1

See Embeddings and Reranking for prefixes, model families, and scoring semantics.

mlxcel serve¶

mlxcel serve [OPTIONS] --model <PATH_OR_REPO_ID>

Starts a persistent HTTP server compatible with the OpenAI API and llama.cpp server API. The server loads the model once and keeps it resident in memory, handling multiple requests through continuous batching.

Several flags accept environment variables for deployment scenarios where command-line arguments are inconvenient.

Network and Auth¶

Concurrency and Batching¶

When --ctx-size C is nonzero, mlxcel follows llama.cpp server semantics: the value is a total server context budget. The effective per-slot context window is floor(C / active_slots), where active_slots is --max-batch-size when set, otherwise --n-parallel; --no-batch uses one slot. /health.context_size, /slots[].context_size, and memory preflight report and use the effective per-slot value. Startup rejects explicit budgets that leave a slot below 512 tokens.

Audio Workers¶

See OpenAI Audio API for Whisper and Kokoro setup.

Embedding and Reranking Workers¶

An embedding or sequence-classifier checkpoint in -m starts that specialized worker without chat. A chat model in -m can run with both secondary providers. A generative reranker-only server names the same checkpoint in -m and --reranker-model; this does not load two copies. One process supports at most one embedding provider and one reranker provider. See Embedding and Reranking APIs.

Speculative Decoding (serve)¶

Diffusion Options (serve)¶

These flags tune block-diffusion generation when the served model is a diffusion model such as DiffusionGemma. Autoregressive models ignore them. See Block Diffusion.

Image Model (serve)¶

Chat Template¶

Endpoint Toggles¶

Model Resolution and Store¶

--hf-file, --model-url, and --docker-repo are parsed for llama-server CLI compatibility but rejected at startup. --hf-file selects one GGUF file inside a repository while MLX loads a whole SafeTensors snapshot, --model-url resolves by URL while mlxcel resolves by repository identifier, and Docker Hub model repositories distribute GGUF artifacts.

Router Mode¶

These flags apply when the server hosts more than one model.

WebUI¶

See WebUI for the security rules and the screens. The llama-server options that configure its own browser UI (--ui-config, --ui-config-file, --path, --tools, --tools-runtime, --mcp-servers-config, --mcp-servers-json, --ui-mcp-proxy, --agent) are refused at startup.

LoRA Adapters (serve)¶

Structured Output Defaults¶

Server-wide constraints applied when a request does not carry its own. A request that specifies its own grammar or schema overrides these.

Responses API Stores¶

These flags control the in-memory stores used by /v1/responses.

Prompt Cache and APC¶

Prompt-prefix KV caching is enabled by default. These flags tune cache capacity and the default-on APC block-hash layer. See Prompt Cache for behavior, metrics, and troubleshooting details.

Default Sampling Parameters¶

These flags set server-wide defaults used when a request does not specify its own sampling parameters. Individual requests can override all of these values.

--adaptive-target and --adaptive-decay drive adaptive-p, which runs only when the sampler list names adaptive_p, as in b10621. --ignore-eos suppresses end-of-generation tokens so generation runs to the token budget or a stop string. --logit-bias biases a token id in every request (TOKEN_ID(+/-)BIAS, for example 15043+1) and is repeatable. --reverse-prompt adds a server-wide stop string merged into every request's stop set and is repeatable. --samplers and --sampler-seq (the single-character b10621 form) are accepted, but mlxcel's chain order is fixed to b10621's default and any other order is rejected at startup.

Distributed Inference¶

Current tp_size > 1 runtime notes:

• Supported families: dense Llama, dense Qwen2/2.5, dense Qwen3, dense Qwen3.5 text, Gemma 3 text, Gemma 4 text, ERNIE 4.5, Hunyuan v1 Dense
• The current runtime is in-process multi-rank on one machine
• LoRA adapters are not supported with TP
• mlxcel serve keeps continuous batching enabled for all currently supported dense TP runtimes except Gemma 4 E2B-style conservative fallback checkpoints
• mlxcel serve defaults decode storage to auto; Llama, Qwen 3, Qwen 3.5 text, Gemma 3 text, and Llama 4 workers select paged decode automatically. Families such as ExaOne4, Ministral3, Gemma 4, Qwen3 Next, GPT-OSS, and standard external-KV workers without model-specific paged decode dispatch remain on dense decode until they satisfy the paged-default rollout checklist in Continuous Batching

Example:

mlxcel generate \
  --model ./models/ernie-4.5-0.3b-4bit \
  --prompt "Write one short sentence about TP." \
  --tp-size 2

Remote TCP pipeline-parallel example:

CLUSTER_NAME=studio-pp \
TRANSPORT_BACKEND=tcp \
COORDINATOR_CONTROL_ADDR=192.168.1.22:19000 \
STAGE0_ADDR=192.168.1.22:19001 \
STAGE1_ADDR=192.168.1.24:19001 \
scripts/benchmark_pipeline_remote_rollout.sh write-config \
  examples/distributed/generated_pipeline_remote_2node_tcp.toml

# Machine B
mlxcel serve \
  --model ./models/llama-3.2-1b-4bit \
  --distributed-config examples/distributed/generated_pipeline_remote_2node_tcp.toml \
  --node-id stage-1 \
  --host 0.0.0.0 \
  --port 18081 \
  --no-warmup

# Machine A
mlxcel serve \
  --model ./models/llama-3.2-1b-4bit \
  --distributed-config examples/distributed/generated_pipeline_remote_2node_tcp.toml \
  --node-id stage-0 \
  --host 0.0.0.0 \
  --port 18081 \
  --no-warmup

# Machine A coordinator
mlxcel serve \
  --model ./models/llama-3.2-1b-4bit \
  --alias llama-remote-pp \
  --distributed-config examples/distributed/generated_pipeline_remote_2node_tcp.toml \
  --node-id coordinator \
  --host 0.0.0.0 \
  --port 18080 \
  --n-parallel 2 \
  --max-batch-size 2 \
  --pp-micro-batch-size 2 \
  --metrics \
  --no-warmup

Smoke test:

COORDINATOR_URL=http://192.168.1.22:18080 \
MODEL_ALIAS=llama-remote-pp \
scripts/benchmark_pipeline_remote_rollout.sh smoke

Thunderbolt mode uses the same workflow with:

• TRANSPORT_BACKEND=thunderbolt
• Thunderbolt Bridge IPs instead of LAN IPs
• the same coordinator/stage split and ports

Example:

CLUSTER_NAME=studio-pp \
TRANSPORT_BACKEND=thunderbolt \
COORDINATOR_CONTROL_ADDR=169.254.20.10:19000 \
STAGE0_ADDR=169.254.20.10:19001 \
STAGE1_ADDR=169.254.20.11:19001 \
scripts/benchmark_pipeline_remote_rollout.sh write-config \
  examples/distributed/generated_pipeline_remote_2node_thunderbolt.toml

Logging¶

Serve Examples¶

Minimal server on localhost:

mlxcel serve --model ./models/Qwen2.5-7B-Instruct

The server starts on 127.0.0.1:8080. Test it with curl:

curl -s http://localhost:8080/health

{"status":"ok"}

Chat completion request:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen2.5-7B-Instruct",
    "messages": [
      {"role": "user", "content": "What is 2 + 2?"}
    ],
    "max_tokens": 50
  }'

Server with authentication and public binding:

mlxcel serve \
  --model ./models/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8080 \
  --api-key "my-secret-key"

Authenticated request:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer my-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"Hello"}]}'

High-concurrency server with batching and metrics:

mlxcel serve \
  --model ./models/Qwen2.5-7B-Instruct \
  --max-batch-size 8 \
  --prefill-chunk-size 512 \
  --enable-preemption \
  --metrics

An explicit --prefill-chunk-size pins the chunk in both scheduler states; omit it to keep the adaptive default (2048 with no live decode, 512 with one).

Server with DFlash speculative decoding:

mlxcel serve \
  --model ./models/qwen3.5-27b-4bit \
  --draft-model ./models/Qwen3.5-27B-DFlash \
  --draft-kind dflash \
  --draft-block-size 16

For the standalone server binary, the primary spelling is --model-draft (--draft-model is also accepted):

mlxcel-server \
  --model ./models/qwen3.5-27b-4bit \
  --model-draft ./models/Qwen3.5-27B-DFlash \
  --draft-kind dflash \
  --draft-block-size 16

Unix socket mode (for local IPC):

mlxcel serve \
  --model ./models/Gemma-3-4B-Instruct \
  --host /tmp/mlxcel.sock

Using environment variables (deployment pattern):

export LLAMA_ARG_MODEL=./models/Qwen2.5-7B-Instruct
export LLAMA_API_KEY=my-secret-key
export LLAMA_ARG_PORT=9000
mlxcel serve

mlxcel list¶

mlxcel list [OPTIONS]
mlxcel ls               # alias

Lists the models downloaded into the local mlxcel store, one row per snapshot with repo-id, on-disk size, and last-modified time. No model loading occurs. To list the supported model architectures instead, use mlxcel arch.

mlxcel list                              # default table, sorted by repo-id
mlxcel list --sort size                  # largest models first
mlxcel list -q                           # repo-ids only, pipe-friendly
mlxcel list --json                       # machine-readable
mlxcel list --models-dir /data/mlxcel-models

mlxcel arch¶

mlxcel arch
mlxcel arch --json
mlxcel supported        # alias

Prints all supported model architectures grouped by category, then exits. With --json, writes the same registry as one machine-readable JSON object for recipe and coverage tooling. No model loading occurs.

Example output (abbreviated):

Supported Model Architectures:

Llama:
  - Llama 1/2/3
  - Llama 4 (MoE)

Speech-to-text:
  - Whisper (encoder-decoder ASR)

Text-to-speech:
  - Kokoro (StyleTTS2 + iSTFTNet)

Embedding:
  - BERT / MiniLM encoder
  - Qwen3-VL-Embedding

Reranker:
  - SequenceClassifier

Other VLM:
  - Falcon-OCR (early fusion)
  - Florence-2 (DaViT + BART seq2seq, task prompts)
  ...

The JSON form reports mlxcel_version plus one families entry per supported architecture with stable id, display_name, category, model_types, aliases, runtimes, modalities_in, output, backends (Metal, CUDA, and ROCm status; every family's ROCm entry is partial while the backend is experimental), tensor_parallel, pipeline_parallel, drafters, and kv_modes fields. The committed recipes/registry/CURRENT marker points at the matching snapshot under recipes/registry/.

The exact list is generated from ALL_MODEL_TYPES in the running binary. Use mlxcel arch rather than a copied list when checking a deployment. See Supported Models for architecture keys and capability notes.

Model detection is automatic. Generation, embedding, and reranking loaders use model_type together with architecture and checkpoint-layout metadata, so no manual architecture flag is needed.

mlxcel download¶

mlxcel download [OPTIONS] <REPO_ID>

Downloads a HuggingFace repository snapshot without Python dependencies. By default, the snapshot lands in the global mlxcel store at ${MLXCEL_CACHE_DIR:-$HOME/.cache/mlxcel}/models/<owner>/<name>, and an existing HuggingFace cache copy is reused when possible. The standalone mlxcel-server binary exposes the same download subcommand for deployment images that ship only the server executable; mlxcel-server download --help (also -h and --usage) prints its help.

Examples:

# Default: global store.
mlxcel download mlx-community/Qwen3-4B-4bit

# Write to an exact directory instead of the store.
mlxcel download mlx-community/Qwen3-4B-4bit --local-dir /tmp/qwen

# Relocate the store root for this command.
mlxcel download mlx-community/Qwen3-4B-4bit --models-dir /data/mlxcel-models

# Gated/private repository.
mlxcel download meta-llama/Llama-3.1-8B-Instruct --token "$HF_TOKEN"

mlxcel rm¶

mlxcel rm [OPTIONS] <REPO_ID>

Removes a downloaded model snapshot from the mlxcel store. It only deletes snapshots under the managed store root; models that exist solely in the read-only HuggingFace cache are reported but never deleted.

Examples:

mlxcel rm mlx-community/Qwen3-4B-4bit
mlxcel rm mlx-community/Qwen3-4B-4bit --yes
mlxcel rm mlx-community/Qwen3-4B-4bit --models-dir /data/mlxcel-models

mlxcel tune¶

mlxcel tune [OPTIONS]

Profiles supported kernel launch configurations and writes the fastest stable choices under ${MLXCEL_CACHE_DIR:-$HOME/.cache/mlxcel}/autotune. Tuning is inert until MLXCEL_AUTOTUNE=cache or MLXCEL_AUTOTUNE=1 is set. See Performance Tuning.

mlxcel tune --dry-run
mlxcel tune --op paged-decode-splits -m ./models/qwen3-4b-4bit
MLXCEL_AUTOTUNE=cache mlxcel serve -m ./models/qwen3-4b-4bit

mlxcel split-mtp¶

mlxcel split-mtp -m <RAW_CHECKPOINT> -o <DRAFTER_DIR> [OPTIONS]

Writes the GLM-4.7-Flash next-token-prediction layer out of a raw zai-org/GLM-4.7-Flash checkpoint into a standalone glm4_moe_lite_mtp drafter directory. The community 4-bit conversions drop the model.layers.47.* tensors, so the source must be the raw checkpoint; the tool reads only the shards that the safetensors index names for that layer. The resulting directory pairs with any glm4_moe_lite target of the same geometry through mlxcel generate --draft-model or mlxcel-server --model-draft. The command needs the surgery Cargo feature, which is on by default.

Bits, group size, and the output path are validated before the output directory is touched.

mlxcel split-mtp -m models/glm-4.7-flash-bf16 -o models/glm-4.7-flash-mtp-4bit --q-bits 4
mlxcel generate -m models/glm-4.7-flash-4bit --draft-model models/glm-4.7-flash-mtp-4bit -p "Hi"

mlxcel prepare-external-ple¶

mlxcel prepare-external-ple -m <CHECKPOINT> -o <VIEW_DIR> [OPTIONS]

Builds a qwen4_exp model view whose PLE n-gram table is read from disk through text_config.ple_storage instead of living in the MLX parameters, which takes about 30 GiB out of the load. The command needs no Python and writes the manifest in the mlx-vlm v2 format. Shards are hard-linked, so the output must be on the same filesystem as the source, and names that would escape the source are refused.

The command prints the target path, row count, table size, and the number of shards linked. On the OptiQ 2-bit checkpoint, loading the resulting view leaves 45.39 GB resident. See External Weight Stores.

mlxcel prepare-expert-stream¶

mlxcel prepare-expert-stream -m <CHECKPOINT>

Writes the contiguous expert file that SSD expert streaming uses automatically when it is present, and verifies it against the checkpoint before reporting. The file goes inside the model directory. The checkpoint must be qwen4_exp. Streaming itself is selected at run time by MLXCEL_EXPERT_STREAM (auto by default) and sized by MLXCEL_EXPERT_ARENA_BYTES; see External Weight Stores.

mlxcel prepare-expert-stream -m models/Qwen3.8-Flash-Next-OptiQ-2bit