Quickstart¶

This guide walks you through installing mlxcel with Homebrew, running a model, starting the server, and querying it through the REST API. The examples use the new repo-id-aware model resolver, so the explicit download step is optional: mlxcel can auto-download a HuggingFace MLX checkpoint on first use and reuse it afterward from any working directory.

Prerequisites¶

Before proceeding, make sure you have:

• Homebrew installed on an Apple Silicon Mac. The tap publishes macOS Apple Silicon builds only; on Linux, use the mlxcel-linux-*-cuda13 archives from the GitHub release page or build from source.
• mlxcel and mlxcel-server on your PATH. The Homebrew formula installs both commands.

Step 0: Install with Homebrew¶

The Homebrew formula installs both the mlxcel CLI and the standalone mlxcel-server binary on macOS Apple Silicon:

brew tap lablup/tap
brew install lablup/tap/mlxcel

Linux builds are CUDA-specific and are not in the tap. They ship as mlxcel-linux-*-cuda13 archives on the GitHub release page; alternatively, build from source. Windows builds are source-only: NVIDIA CUDA is experimental, and AMD ROCm has been supported since 0.8.

Verify that both commands are available:

mlxcel --help
mlxcel-server --help

If you build from source instead, use ./target/release/mlxcel and ./target/release/mlxcel-server in place of the installed commands in the examples below. See Building from Source for source build prerequisites.

Step 1: Run a Model¶

mlxcel works directly with MLX-format HuggingFace checkpoints. The quickest path is mlxcel run: pass a HuggingFace owner/name repo-id, a bare MLX Community model name, or a local model directory.

This guide uses Qwen3.5 0.8B in 4-bit quantization as an example. It is small enough for a first run while still exercising the same resolver, downloader, generation, and server paths used by larger models.

# Interactive chat REPL. Auto-downloads into the mlxcel store on first use.
mlxcel run mlx-community/Qwen3.5-0.8B-4bit

# Bare names resolve to mlx-community/<name> by default.
mlxcel run Qwen3.5-0.8B-4bit

# One-shot generation with -p, then exit.
mlxcel run Qwen3.5-0.8B-4bit -p "Explain attention in one sentence." -n 100

# No model argument falls back to mlx-community/gemma-4-e2b-it-4bit.
mlxcel run

mlxcel run is a client of the server engine: it starts the mlxcel-server model worker in-process (one slot, no HTTP listener) and sends each turn through the /v1/chat/completions request path. mlxcel run -p and mlxcel-server therefore give the same output for the same request, and run -p prints only the reply, without echoing the prompt. See mlxcel run for details.

The same one-command flow works for larger checkpoints. For example, this starts a Gemma 4 chat session and downloads mlx-community/gemma-4-e4b-it-4bit into the store if it is not already present:

mlxcel run gemma-4-e4b-it-4bit

For a smoke test that exits automatically, add a prompt and a small token limit:

mlxcel run gemma-4-e4b-it-4bit -p "Hello!" -n 8 --temp 0

Example output from an Apple Silicon machine with the model already present locally:

Runtime device: Apple GPU (Metal)
Wired memory limit: 107.5 GB
[mlxcel] 'gemma-4-e4b-it-4bit' -> mlx-community/gemma-4-e4b-it-4bit
Loading model from "models/gemma-4-e4b-it-4bit"...
Loaded Gemma4 audio encoder (12 Conformer layers)
Model loaded in 1.440s (resident: 4.83 GB, peak: 4.86 GB).
Generating...
Hello!...

[Generated 8 tokens in 0.17s = 46.62 tok/s]

On a clean machine, the same command prints download progress before Loading model from ...; timings and generated text vary by hardware and cache state.

With no -p/--prompt, mlxcel run opens an interactive multi-turn chat REPL. Use /bye to exit, /clear to reset the conversation, /? for help, and triple quotes (""") for multiline input.

What is MLX format? MLX models store weights in SafeTensors files (.safetensors) alongside a config.json describing the architecture and tokenizer metadata such as tokenizer_config.json. mlxcel loads these files directly without a conversion step.

Step 2: Generate, Inspect, or Preflight¶

generate, serve, and inspect accept the same model argument through -m/--model:

• an existing local path,
• a HuggingFace owner/name repo-id, or
• a bare model name resolved as mlx-community/<name>.

On a cache miss, repo-id arguments are downloaded into the global mlxcel store and reused afterward.

# One-off text generation.
mlxcel generate \
    --model Qwen3.5-0.8B-4bit \
    --prompt "Explain the concept of attention in transformer models." \
    --max-tokens 200

# Read-only memory budget: weights + KV cache + runtime headroom.
mlxcel inspect --model Qwen3.5-0.8B-4bit --max-tokens 32768

# Refuse to start if the estimated model + KV cache will not fit.
mlxcel generate \
    --model Qwen3.5-0.8B-4bit \
    --prompt "Hello, world!" \
    --max-tokens 32768 \
    --estimate-memory

Use --force (alias: --no-memory-check) only after reviewing an over-budget preflight. Set MLXCEL_MEMORY_LIMIT=32GB or a similar value when you want the estimator to honor a soft deployment budget instead of all available system memory.

CLI Options at a Glance¶

Optional: Add Faster Decoding¶

Start by timing the target alone with --profile. If you have a smaller model that shares the target's tokenizer, add speculative decoding with that model as a full-model drafter to a one-shot generation:

mlxcel generate \
    --model ./models/target-model \
    --draft-model ./models/smaller-compatible-model \
    --draft-kind full-model \
    --draft-block-size 5 \
    --prompt "Explain speculative decoding in three sentences." \
    --max-tokens 200 \
    --profile

Without a second model, mlxcel generate --prompt-lookup copies earlier text from the context as draft tokens, and mlxcel serve --draft-kind prompt-lookup does the same inside the server's batch scheduler. Both pay off mainly on replies that repeat the prompt.

Target-specific assistants use a specialized server path. For example, pair a Gemma 4 target with its published assistant and select MTP:

mlxcel serve \
    --model ./models/gemma-4-target \
    --draft-model ./models/gemma-4-assistant \
    --draft-kind mtp \
    --draft-block-size 4 \
    --max-batch-size 1 \
    --port 8080

For a supported Qwen 3.5-family target and its matching DFlash checkpoint, change the kind and width:

mlxcel serve \
    --model ./models/qwen3.5-target \
    --draft-model ./models/qwen3.5-dflash \
    --draft-kind dflash \
    --draft-block-size 16 \
    --max-batch-size 1 \
    --port 8080

The target and drafter must be an explicitly compatible pair, both models consume memory, and a configured path can decline or fall back when a request is not qualified. Compare target-only latency and output before keeping the setting. See Speculative Decoding for the algorithm and MTP and DFlash for supported pairings, exactness gates, multimodal limits, and tuning.

Step 3: Start the OpenAI-Compatible Server¶

The serve subcommand starts an HTTP server that exposes OpenAI-compatible endpoints. Applications that already work with OpenAI-style APIs can usually switch to mlxcel with a base URL change.

mlxcel serve --model Qwen3.5-0.8B-4bit --port 8080

You should see output similar to:

Loading model: mlx-community/Qwen3.5-0.8B-4bit
Warming up model...
Server listening on http://127.0.0.1:8080

The server performs a warmup pass on startup (generating one token and discarding it) to pre-compile the Metal pipeline. This adds a few seconds to startup but eliminates the first-request latency spike.

Alternatively, use the standalone mlxcel-server binary, which is designed as a drop-in replacement for llama.cpp's llama-server. Its -m/--model argument accepts the same local path, repo-id, and bare-name forms as mlxcel serve:

mlxcel-server -m Qwen3.5-0.8B-4bit --port 8080

Step 4: Query the Server¶

With the server running, send requests using curl or any HTTP client. The server accepts OpenAI-format chat completion requests.

Non-Streaming Request¶

curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen3.5-0.8B-4bit",
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "max_tokens": 100,
        "temperature": 0.7
    }'

Response:

{
    "id": "chatcmpl-abc123",
    "object": "chat.completion",
    "model": "Qwen3.5-0.8B-4bit",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "The capital of France is Paris."
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 15,
        "completion_tokens": 8,
        "total_tokens": 23
    }
}

Streaming Request¶

For real-time token streaming, set "stream": true. The server responds with Server-Sent Events (SSE):

curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen3.5-0.8B-4bit",
        "messages": [
            {"role": "user", "content": "Write a haiku about Rust programming."}
        ],
        "stream": true
    }'

Each line of the response is a data: event containing a JSON delta:

data: {"id":"chatcmpl-abc123","choices":[{"delta":{"content":"Memory"},"index":0}]}
data: {"id":"chatcmpl-abc123","choices":[{"delta":{"content":" safe"},"index":0}]}
...
data: [DONE]

The server also exposes /v1/responses for the OpenAI Responses API, /v1/messages for the Anthropic Messages API, /v1/embeddings, /v1/rerank, native /completion and /embedding, and OpenAI-compatible speech and transcription routes under /v1/audio. See the REST API reference, Embedding and Reranking APIs, and Audio API guide for exact schemas.

Step 5: Embed and Rerank¶

Create and compare vectors offline with an embedding checkpoint:

mlxcel embed -m sentence-transformers/all-MiniLM-L6-v2 \
    -p "Vector search retrieves nearby embeddings." \
    -p "A compiler translates source code."

Score candidate documents against one query with a cross-encoder reranker:

mlxcel rerank -m BAAI/bge-reranker-v2-m3 \
    -q "What is vector search?" \
    -d "Vector search retrieves nearby embeddings." \
    -d "A compiler translates source code." \
    --top-n 1

For HTTP serving, put chat in -m and add the two specialized providers. Each owns a dedicated worker thread:

mlxcel serve \
    -m mlx-community/Qwen3-4B-4bit \
    --embedding-model Qwen/Qwen3-Embedding-0.6B \
    --reranker-model mlx-community/Qwen3-Reranker-0.6B-4bit \
    --port 8080

Use /v1/embeddings for the OpenAI envelope and /v1/rerank for Jina/Cohere-style results. The native /embedding and /embeddings paths return a different llama-server response shape. Prefixes, pooling, and normalization are part of the model contract; read Embeddings and Reranking before building an index.

Step 6: Manage Downloaded Models¶

Downloaded models land in a location-independent global store:

${MLXCEL_CACHE_DIR:-$HOME/.cache/mlxcel}/models/<owner>/<name>

List and prune the store from any directory:

# List downloaded models with their on-disk size and absolute path.
mlxcel list

# Remove a model from the mlxcel store (prompts for confirmation).
mlxcel rm mlx-community/Qwen3.5-0.8B-4bit

# Remove without the prompt for scripts.
mlxcel rm mlx-community/Qwen3.5-0.8B-4bit --yes

To relocate the store, set MLXCEL_MODELS_DIR or pass --models-dir. The override is the directory that directly contains <owner>/<name> snapshots; it does not add another models/ component.

MLXCEL_MODELS_DIR=/data/mlxcel-models mlxcel list
mlxcel download mlx-community/Qwen3.5-0.8B-4bit --models-dir /data/mlxcel-models

Use --local-dir with mlxcel download only when you want to opt out of the global store and write a snapshot to one exact directory.

Step 7: Using Vision-Language Models¶

mlxcel supports Vision-Language Models (VLMs) that can process images alongside text. To use a VLM:

1. Let the resolver auto-download a VLM from the MLX Community, or download it explicitly:

mlxcel download mlx-community/Qwen2-VL-7B-Instruct-4bit

1. Generate text with an image using the CLI:

mlxcel generate \
    --model mlx-community/Qwen2-VL-7B-Instruct-4bit \
    --prompt "Describe what you see in this image." \
    --image ./path/to/image.jpg \
    --max-tokens 200

1. Send image requests to the server using the OpenAI vision format (base64-encoded image or URL):

curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen2-VL-7B-Instruct-4bit",
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "What is in this image?"},
                    {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
                ]
            }
        ]
    }'

Representative VLM families include Gemma 3/3n/4, LLaVA, Llama Vision/4, Florence-2, Falcon-OCR, Jina VLM, MiniCPM-o/V, Molmo, Muse Glimmer, Phi-3 Vision, Phi-4 Multimodal, Qwen2/2.5/3/3.5-VL, and Youtu-VL. The complete maintained matrix is in Supported Models.

Next Steps¶

• Read the Configuration guide to tune model-store paths, memory limits, and device selection.
• Explore the CLI reference for all available options.
• See the REST API reference for the full endpoint specification.
• Use the Embedding and Reranking API for retrieval request and response contracts.
• Use the OpenAI Audio API guide for Whisper transcription/translation and Kokoro speech synthesis.
• Run Performance Tuning when you need hardware-specific paged-decode autotuning.
• Learn about distributed inference for multi-GPU setups.