Supported Models¶

mlxcel detects the model architecture automatically from checkpoint metadata. The model_type field in config.json is the primary generation-family key; architectures, sentence-transformers pooling layouts, and classifier heads additionally distinguish embedding and reranker checkpoints. Run mlxcel arch to inspect the exact matrix compiled into a particular binary.

Browse mlxcel Recipes for model-specific commands and hardware guidance; each supported registry family also has a page at https://mlxcel.ai/en/recipes/families/<id>/.

Current Tensor Parallel Runtime¶

The current tp_size > 1 runtime is available for the following dense text families:

Current TP constraints:

• --tp-embedding-mode replicated
• --tp-lm-head-mode replicated
• No LoRA adapters with TP
• TP server batching is enabled for dense Llama, Qwen2/2.5, Qwen3, Qwen3.5, Gemma 3 text, Gemma 4 text, ERNIE 4.5, and Hunyuan v1 Dense
• Server decode storage defaults to automatic selection; paged decode is selected by default for Llama, Qwen 3, Qwen 3.5 text, Gemma 3 text, and Llama 4 workers, while other families stay on dense decode unless explicitly opted in
• Gemma 4 E2B-style checkpoints (num_key_value_heads=1 with per-layer inputs + KV sharing) currently run through a conservative fallback and force sequential server execution

Real-model CUDA parity coverage currently includes:

• tp=2: Qwen2.5 0.5B (24-token greedy parity)
• tp=2: Llama, Qwen 3, Qwen 3.5 0.8B and 9B, Gemma 3, Gemma 4 E2B (generation smoke), ERNIE 4.5, Hunyuan v1 Dense
• tp=4: Qwen2.5 7B
• tp=4: Llama, Qwen 3, Qwen 3.5 4B, 9B, and 27B, Gemma 3, Gemma 4 31B / E4B / 26B-A4B, ERNIE 4.5, Hunyuan v1 Dense
• Additional Gemma 4 TP smoke coverage: gemma-4-e2b-it-8bit, gemma-4-e4b-it-8bit, gemma-4-31b-it-4bit, gemma-4-31b-it-nvfp4 (F8_E4M3 weight scales converted to f16 at load time)

Table of Contents¶

• Standard Transformer Models
• Mixture-of-Experts (MoE) Models
• SSM and RNN Models
• Hybrid Models
• Vision-Language Models (VLM)
• Regional and Specialized Models
• Audio Models
• Embedding Models
• Reranker Models
• Object Detection Models
• Dense Prediction and Token-Level Models, including image generation
• Code Models
• Model Detection

Standard Transformer Models¶

Mixture-of-Experts (MoE) Models¶

SSM and RNN Models¶

Hybrid Models¶

Hybrid models combine multiple architectural paradigms (e.g., Transformer attention layers with SSM or linear-attention layers).

AI21-Jamba-Reasoning-3B starts every reply in reasoning_content; the answer reaches content once the model writes </think>, and a max_tokens budget that ends first returns an empty content with finish_reason: "length". Short factual answers take 150 to 450 completion tokens at temperature 0. In multi-turn chats, echo reasoning_content back with each assistant turn, or rely on the server's reasoning re-echo while the prompt cache is on; otherwise the chat template renders earlier turns differently and the prompt cache misses.

Vision-Language Models (VLM)¶

VLMs are automatically distinguished from their text-only counterparts by the presence of a vision_config section in config.json.

A checkpoint with a VLM model_type loads text-only when vision_config is absent, null, or {}, when language_model_only is true, or when its weights carry no vision tower (vision_tower.*, model.visual.*, and similar prefixes). Text requests run normally; an image or video request fails, on the Qwen-VL families with model <path> was loaded without a vision tower (text-only checkpoint). For Qwen 3.5, language_model_only: true wins even when vision weights ship (a notice goes to stderr), and a non-boolean value is rejected at load.

Audio Models¶

Audio-only checkpoints do not use the text-generation worker. Start a dedicated server and use the OpenAI Audio API, or for Nemotron VoiceChat the Realtime VoiceChat API.

Regional and Specialized Models¶

Embedding Models¶

Embedding checkpoints run through mlxcel embed or the embedding server routes rather than text generation. Detection checks encoder-only model_type values, known embedding architectures, a sentence-transformers Pooling module, or 1_Pooling/config.json. See Embedding and Reranking APIs for serving and Embeddings and Reranking for the model contracts.

Dense Text Embeddings¶

Multimodal and Late-Interaction Embeddings¶

Qwen3-VL-Embedding, Llama-Nemotron-VL-Embed, and EmbeddingGemma 2 return a single vector for an image; an EmbeddingGemma 2 checkpoint without a vision tower, such as embedding_gemma2_text, refuses image inputs. ColIdefics3 and ColQwen2.5 return a token-vector matrix and must be ranked with MaxSim. A LoRA-only late-interaction repository is rejected; merge the adapter into a complete checkpoint first. The SigLIP embedding family exposes only its text tower.

Reranker Models¶

Rerankers are served through mlxcel rerank or any of the reranking routes. All supported kinds return a relevance score in [0, 1].

Only one-label sequence classifiers can be identified as rerankers from -m alone. A generative checkpoint is indistinguishable from an ordinary generator at detection time, so the explicit secondary flag selects its ranking semantics. Qwen3-VL supports image-bearing queries and documents, processes image rows individually, and does not rerank video.

Object Detection Models¶

Object-detection models do not emit a token stream. Use the dedicated mlxcel detect command, which loads the checkpoint, runs the image processor, and prints bounding boxes in text or JSON format.

Example:

mlxcel detect -m mlx-community/docling-layout-heron-mlx-bf16 -i page.png --format json
mlxcel detect -m axiom-of-choice/OmniParser-v2-icon-detect -i ui.png --size 1280 --format json

Dense Prediction and Token-Level Models¶

Three further families of checkpoints do not produce a chat token stream and each has its own subcommand.

Image-generation checkpoints are a fourth surface (mlxcel generate-image, and /v1/images/generations and /v1/images/edits behind --image-model). The families are FLUX.2 Klein (flux2), Z-Image (z_image), Prism Bonsai ternary FLUX.2 Klein (bonsai), ERNIE-Image (ernie_image), Mage-Flow (mage_flow), Qwen-Image-2.1 (qwen_image), Ideogram 4 (ideogram4), Ming-Image-0.1-Design (ming_image), and LLaDA-Image (llada_image). Image Generation lists checkpoints, tasks, and default steps.

Page OCR pipelines load a whole document through mlxcel generate rather than through a subcommand. IndicOCR (indic_ocr) pairs a PP-DocLayoutV3 layout stage with a Qwen3.5-0.8B block transcriber for English and 22 Indian languages, and returns Markdown in reading order or a block JSON file:

mlxcel generate -m HashNuke/indic-ocr-mlx --image page.png --output-json page.json

--table-format markdown renders tables as GitHub-flavored Markdown instead of HTML, and -n/--max-tokens overrides the 2048-token per-block default. A prompt is not needed because each region's instruction comes from its layout class. --layout-detections FILE replays detection JSON from mlxcel detect instead of running the layout stage.

Code Models¶

Model Detection¶

Architecture detection is performed in src/models/detection.rs. Generation families primarily match config.json::model_type; task-kind detection first checks embedding layouts and supported one-label sequence-classification heads so they are not misrouted to generation.

For models that have both text-only and vision-language variants (Gemma 3, Llama 4, Qwen 3.5, Mistral 3, Gemma 3n), detection is automatic:

if config.json contains "vision_config" key
    => load as VLM
else
    => load as text-only model

Florence-2 is an encoder-decoder family. It is detected the same way, from model_type: florence2, but it does not join the shared autoregressive loop: mlxcel generate drives its own seq2seq pipeline and mlxcel-server serves it through a dedicated single-stream worker rather than the batched or paged scheduler. Nemotron-Parse (model_type: nemotron_parse) follows the same pattern.

For the Hunyuan family, detection is based on num_experts:

if num_experts > 1  => HunyuanMoe
else                => HunyuanV1Dense

Embedding detection also recognizes encoder-only families, known embedding architecture names, a sentence-transformers Pooling module, or 1_Pooling/config.json. A supported ForSequenceClassification architecture always selects reranking before those embedding rules. Generative Qwen3 and Qwen3-VL rerankers require --reranker-model because their configuration is an ordinary generation configuration.

Quantization Formats¶

Quantization is read from the quantization block of config.json. Besides affine 4-bit and 8-bit and the MXFP4, MXFP8, and NVFP4 modes, mlxcel loads affine 1-bit checkpoints (bits: 1, group_size 32, 64, or 128, with .scales and .biases): Metal runs them with fused 1-bit kernels, and CUDA and ROCm run the dequantize graph. Only dense linears and embeddings may be 1-bit; 1-bit MoE expert stacks, 1-bit MLA per-head projections, and 1-bit planes without .biases are refused at load. MLXCEL_ONE_BIT_KERNEL=0 forces the dequantize graph on Metal. Validated on prism-ml/Bonsai-1.7B-mlx-1bit and prism-ml/Bonsai-8B-mlx-1bit (qwen3). See Quantization Support for per-backend behavior.

The fine-grained FP8 hub builds (quantization_config: {"quant_method": "fp8", "fmt": "e4m3", "weight_block_size": [128, 128]}) and llm-compressor's compressed-tensors float-quantized exports both load: mlxcel reconstructs raw E4M3 bytes with their block or channel scale and requantizes each tensor to MLX-native mxfp8, one tensor at a time. Compressed-tensors accepts per-output-channel and block scales; strategy: "tensor" or "group", another num_bits, and dynamic weights are named load errors. Validated on RedHatAI/Qwen3.5-4B-FP8-dynamic against its bf16 base; the per-channel scale is load-bearing, and traces without it produce non-finite logits.

Adding a New Model¶

See the Adding Models guide for the complete checklist.