Architecture Overview¶

Table of Contents¶

• What mlxcel Is
• High-Level Runtime Diagram
• Control Plane vs Data Plane
• Key Design Principles
• Zero Python Dependency
• Config-Driven Model Detection
• Enum-Based Exhaustive Dispatch
• The LanguageModel Trait
• Modular VLM Pipeline
• Module Map
• Two Binaries
• Environment Configuration

What mlxcel Is¶

mlxcel is a high-performance, general-purpose inference engine written in Rust. Its MLX backend runs generation, multimodal understanding, audio (including Nemotron VoiceChat full-duplex speech), detection, embeddings, and reranking on Apple Silicon through Metal, on supported NVIDIA GPUs through CUDA, and experimentally on AMD GPUs through ROCm; no Python runtime sits in the inference request path.

The key architectural commitment is a direct binding from Rust to MLX C++, bypassing Python entirely. Weight loading, forward passes, sampling, pooling, normalization, ranking, and dedicated audio, detection, and diffusion runtimes all reach the native MLX computation graph through Rust APIs.

High-Level Runtime Diagram¶

The diagram below shows the common path from command or HTTP entry points through model-kind detection, task-specific workers, native tensor operations, and the MLX runtime. Generation uses a batch or sequential scheduler; embedding and reranking use dedicated MLX-owning workers; block diffusion, audio, sequence-to-sequence vision, and detection also use specialized runtimes.

CLI commands resolve a checkpoint, select a task, and invoke the matching forward path. The server can coordinate concurrent generation through its batch scheduler while an embedding worker and reranking worker separately own their model, tokenizer, and MLX arrays. These task paths share the same native MLX binding and common tensor primitives on Metal, CUDA, ROCm, or CPU.

Control Plane vs Data Plane¶

mlxcel draws a clear boundary between the control plane and the data plane.

The control plane lives in src/ (the main crate). It is responsible for:

• Reading config.json to identify model architecture
• Selecting and executing the correct loader
• Constructing the LoadedModel enum wrapper
• Loading and managing the tokenizer
• Applying chat templates (Jinja2 via minijinja)
• Installing the load-time model surgery pipeline when configured
• Assembling the SamplingConfig from user-facing parameters
• Routing VLM requests through image preprocessing and embedding projection
• Resolving embedding pooling, normalization, instructions, and length limits
• Constructing reranker pairs, multimodal items, and response envelopes
• Managing the HTTP server, generation scheduler, specialized worker queues, and request lifecycle

For autoregressive generation, the control plane does not implement tensor math itself. It drives the data plane through the LanguageModel trait and the decode Engine in mlxcel-core: mlxcel generate decodes on DirectEngine, a one-sequence client of Engine, and the server's BatchScheduler keeps admission, tick policy, preemption, and prompt-cache lookup while every forward runs through Engine. Embedding, reranking, audio, detection, diffusion, and sequence-to-sequence runtimes use their own typed model or worker interfaces.

The data plane lives in src/lib/mlxcel-core/. It is responsible for:

• Executing autoregressive model forward passes through LanguageModel::forward()
• Running every forward, sampler draw, and finish step through Engine (open, prefill, step, the pipelined submit/finish_rows pair, and close); a single sequence is a step batch of one
• Managing KV cache state through KVCache, RotatingKVCache, and ChunkedKVCache, which also choose dense or paged attention for the model (KVCache::attend, cache::attend_batched)
• Planning prefill geometry with PrefillPlan and sampling each row through RowSampler, with token bias composed by compose_token_bias
• Bridging tensor operations to MLX C++ through the cxx bridge declared in src/lib/mlxcel-core/src/lib.rs and wrapped by modules such as ops.rs
• Running the speculative round loops: the engine's token-only loop (Engine::verify_round) for the prompt-lookup and full-model drafters, and the MTP and DFlash drafters
• Providing common attention-mask, pooling, normalization, projection, and layer operations used by embedding and ranking models

The data plane is tensor-compute focused and can execute on Metal, CUDA, ROCm, or CPU. mlxcel's own fused kernels live in src/lib/mlxcelverse/kernels/ and are selected by the resolved GPU backend; most have Metal, CUDA, and ROCm (HIP) ports, and a backend without a port of a given kernel takes the MLX graph fallback for that step. It has no knowledge of HTTP routing or chat-template policy; model wrappers still encode the architecture-specific tensor graph.

This separation keeps each concern testable in isolation. Sampling logic in sampling.rs can be tested with simple float tensors, the engine in engine/ with stub LanguageModel implementations, and control-plane routing in loading/mod.rs by mocking weight loading.

Key Design Principles¶

1. Zero Python Dependency¶

mlxcel uses cxx to bind directly to MLX C++ without a Python interpreter. The bridge declarations live in src/lib/mlxcel-core/src/lib.rs; Rust wrappers such as ops.rs, layers.rs, and sampling.rs expose typed APIs over the C++ operations used by model runtimes.

This eliminates the Python startup time, Python GIL contention, and Python object overhead. It also means the inference process can be a single native binary with no pip dependencies.

2. Config-Driven Model Detection¶

When you point mlxcel at a model directory, it reads config.json and related checkpoint metadata. get_model_type() in src/models/detection.rs maps known generation keys to ModelType, while architectures, sentence-transformers pooling layouts, bidirectional markers, and sequence-classification heads can select embedding or reranking kinds before ordinary generation dispatch.

The mapping is exhaustive for registered architecture variants: an unknown or contradictory layout fails with a clear error rather than silently choosing the wrong task or architecture.

Some model types require additional inspection. For example, hunyuan checks the num_experts field to distinguish the MoE and dense variants. Models with a vision_config field at the top level are automatically redirected to their VLM variant.

3. Enum-Based Exhaustive Dispatch¶

Loaded generative models are stored in the LoadedModel enum defined in src/loaded_model.rs, spanning local, distributed, text, VLM, SSM/RNN, and hybrid execution paths. Embedding, reranking, audio, and object-detection models use task-specific traits or runtime wrappers rather than forcing every model kind into the generative enum.

The LanguageModel trait is delegated to the inner model through a single delegate_language_model! macro, which generates an exhaustive match arm for every variant. This has two properties:

• Adding a new model variant without updating the dispatch macro causes a compile error, not a silent fallback to wrong behavior.
• The Rust compiler can verify that every trait method is covered for every variant.

The enum approach preserves family identity, which matters for capability dispatch (for example, deciding whether a VLM embedding path is needed).

4. LanguageModel Trait¶

The LanguageModel trait in mlxcel-core/src/generate.rs is the single interface between the control plane and data plane for text generation. It defines:

Most families, including Mamba, Mamba2, Jamba, NemotronH, and RWKV7, return true from supports_batching(): their recurrent or model-owned state is keyed by SequenceId, so several sequences can share one batched forward pass. batched_decode_kind() reports BatchedDecodeKind::Native when the model runs that pass as one shared graph and PerRowFallback when it loops over rows; the server logs decode path: native batched forward or decode path: per-row fallback at startup. A model that returns false (for example the OCR, Florence-2, Nemotron-Parse, Nemotron VoiceChat, and diffusion text runtimes) is clamped to max_batch_size 1 and served one sequence at a time.

5. Modular VLM Pipeline¶

Vision-language models extend the text generation pipeline with three additional stages:

1. Visual Encoder: An image backbone (SigLIP, CLIP, ViT, MobileNetV5, etc.) converts pixel data into visual feature tensors.
2. Projection / Connector: A learned projection maps the visual feature space into the language model's embedding space.
3. Embedding Merger: The projected visual tokens are inserted into the token embedding sequence at positions marked by image placeholder tokens.

The merged embedding sequence replaces the normal input_ids path during the prefill step. After the prefill, the VLM decode step is identical to standard text generation: the model processes one token at a time with the accumulated KV cache.

Each VLM family has its own image preprocessing pipeline and placeholder token convention. These are handled in family-specific modules under src/multimodal/ and src/vision/.

Module Map¶

Two Binaries¶

The mlxcel workspace produces two binaries.

mlxcel is the command-line tool. It supports these user-facing subcommands:

• run: Run a positional model argument in interactive chat mode, or one-shot generation with -p. Repo-id and bare-name arguments auto-download on first use.
• generate (alias gen): Run a generation pass for a prompt, or enter the same interactive REPL when the prompt is omitted. Accepts all model, generation, and sampling options.
• embed: Create text or image embeddings offline and print cosine or MaxSim comparisons.
• rerank: Score documents against a query offline and print their ranking.
• inspect: Print a pre-load memory budget for a local path, repo-id, or bare model name without loading tensors.
• detect: Run RT-DETRv2 object detection on an image and print text or JSON boxes.
• serve: Start the generation, embedding, reranking, audio, and utility HTTP server. Accepts server configuration and task-specific provider options.
• list (alias ls): List downloaded model snapshots in the local store, with --json / -q / -v / --sort output controls.
• arch (alias supported): Print all supported model architectures grouped by family.
• download: Download a HuggingFace repository snapshot into the global mlxcel store by default, or to an exact directory with --local-dir.
• rm: Remove a downloaded model snapshot from the managed mlxcel store.
• tune: Profile supported kernel launch configurations and write an opt-in autotune cache.

mlxcel-server is a dedicated server binary with the same serve functionality but packaged for deployment contexts where the full mlxcel CLI is less relevant. It also exposes the shared download subcommand so server-only deployment images can fetch model snapshots without Python tooling.

Both binaries share all library code. The difference is only in the entry point and argument surface.

Environment Configuration¶

Two foundational runtime variables are summarized here. The complete inventory, including memory limits, scheduler, cache, speculative, embedding, reranking, media, and diagnostic controls, is in Environment Variables.

MLXCEL_DEVICE selects the compute backend. On Apple Silicon, gpu and metal are equivalent and route computation through MLX's Metal backend. Setting cpu forces all computation onto the CPU, which is much slower but useful for debugging.

MLXCEL_WIRED_LIMIT controls how much GPU memory MLX pins (wires) so it cannot be evicted. The default behavior matches Python mlx-lm: the limit is set to the total GPU memory size, ensuring model weights stay resident for large models. Setting 0 or none disables wiring entirely. Explicit values like 24GB allow fine-grained control on machines where multiple processes share GPU memory.