Concepts¶
This section explains the core concepts behind mlxcel's design and implementation. Each page is organized around a concept: what it is, why it exists, and how it is implemented in this codebase.
The pages follow a request from command invocation through model-kind detection and into generation or a specialized inference path.
Pages in this Section¶
Architecture Overview¶
A high-level tour of the entire system. Covers the two-binary structure (mlxcel CLI and mlxcel-server), generation and specialized embedding/reranking workers, the separation between control plane and data plane, and the direct Rust-to-C++ FFI bridge to MLX.
Start here if you are new to the codebase.
Model Loading¶
Explains how mlxcel loads a model from disk. Covers model-kind detection for generation, embedding, and reranking; generation policy routing and loader families; SafeTensors weight loading; and the load-time surgery hook.
Read this page if you want to understand how mlxcel identifies and constructs the LoadedModel wrapper that serves as the central runtime handle.
Inference Pipeline¶
Explains the full text generation lifecycle: from raw user input through tokenization, chat template rendering, the autoregressive generation loop, and token streaming to the final decoded output. Also describes the VLM variant that adds image processing, visual encoding, and cross-modal projection before the text generation loop.
Read this page to understand what happens at inference time, and how the prefill and decode steps differ in compute characteristics.
Audio Input Preprocessing¶
Describes the strict WAV input contract, checkpoint-derived policy limits, Phi-4 Multimodal and Gemma 3n preprocessing, server admission budgets, and cancellation behavior for multimodal audio requests.
KV Cache¶
Explains the key-value cache that makes autoregressive decoding efficient. Separates the three structural attention-cache layouts (standard, rotating, and chunked) from storage/quantization modes and the dense/paged server backend, and explains per-sequence state, recurrent hybrids, and memory accounting.
Read this page to understand the memory and performance tradeoffs in the decode step.
Sampling¶
Explains how mlxcel selects the next token at each decode step. Covers the full probability distribution over the vocabulary, greedy decoding, temperature scaling, Top-K and Top-P (nucleus) sampling, Min-P filtering, and the penalty-based methods: repetition penalty, frequency/presence penalties, DRY (Don't Repeat Yourself), XTC, and p-less sampling. Also briefly describes speculative decoding.
Read this page to understand how sampling configuration parameters interact and affect output quality, diversity, and coherence.