Skip to content

Welcome to mlxcel

mlxcel is a high-performance, general-purpose AI inference engine built in Rust. It serves autoregressive and block-diffusion generation, image/audio/video-conditioned generation, speech recognition and synthesis, full-duplex speech conversation, sequence-to-sequence vision tasks, object detection, dense and multi-vector embeddings, and text or multimodal reranking through native MLX C++ bindings on Apple Silicon GPUs, CUDA on NVIDIA GPUs, and experimental ROCm on AMD GPUs (native Windows builds exist for both GPU vendors), with no Python runtime in the inference path.

Key Highlights

  • Smaller Runtime Surface - Model loading, scheduling, and inference stay in one native Rust server process with no Python environment, interpreter bridge, or subprocess layer.
  • Simple Native Deployment - mlxcel and mlxcel-server build as native executables, which makes packaging, service supervision, and upgrades straightforward once the platform runtime libraries are available.
  • llama-server-Style Operation - mlxcel-server accepts many llama-server-compatible flags and LLAMA_ARG_* environment variables to simplify migration from llama.cpp-based scripts.
  • OpenAI-, Anthropic-, and Retrieval-Compatible Server - The HTTP server supports SSE streaming plus generation, Responses, Messages, Audio, /v1/embeddings, and /v1/rerank, a /v1/realtime WebSocket for full-duplex speech with Nemotron VoiceChat, alongside llama-server-native retrieval and utility endpoints.
  • Bundled WebUI - --webui serves an administrative browser UI from the server binary for browsing, downloading, loading, and chatting with models, with no separate frontend service.
  • Serving Features for Real Deployments - Continuous batching on a native batched decode path for most model families, prompt-prefix caching, Automatic Prefix Caching (APC), speculative decoding (MTP, DFlash, full-model drafters, and drafter-free prompt lookup), and KV-cache compression are available for supported model/runtime combinations.
  • Runtime Weight and Sampling Controls - mlxcel exposes YAML load-time model surgery (scale, add, prune, replace, interpolate) and script-aware language bias policies for adjusting model behavior without retraining or checkpoint conversion.
  • Multi-Device and Distributed Modes - Tensor parallelism and pipeline parallelism are implemented for selected model families, including zero-config pipeline startup with static or mDNS-based discovery.
  • Broad Model-Family Coverage - mlxcel arch reports the exact model matrix built into the running binary across dense, MoE, hybrid, diffusion, VLM, embedding, reranker, object-detection, STT, and TTS runtimes.

How to Use This Documentation

If you are new to mlxcel, start with the Quickstart to install the package and run your first model.

  • Quickstart - Get up and running in under 5 minutes.
  • Concepts - Understand how mlxcel works under the hood.
  • CLI Commands - Run interactive chat, generate text, create embeddings, rerank candidates, inspect memory budgets, start a server, manage model snapshots, detect objects, or profile kernel launch configurations.
  • Advanced Inference and Serving - Study speculative decoding, MTP, DFlash, block diffusion, runtime model surgery, embedding model contracts, and retrieve-then-rerank system design.
  • Distributed Inference - Scale across multiple GPUs.
  • REST API - Integrate mlxcel into your applications.
  • OpenAI Audio API - Serve Whisper transcription/translation or Kokoro speech synthesis.
  • Realtime VoiceChat API - Stream full-duplex speech with Nemotron VoiceChat over the /v1/realtime WebSocket.

mlxcel is developed by Lablup Inc. and open-sourced under the Apache License 2.0.