Configuration¶

mlxcel is configured through a combination of environment variables (which affect all invocations) and command-line flags (which override environment variables for a specific run). This page documents the environment variables that control device selection and memory management, as well as the server-specific configuration options.

Table of Contents¶

• Environment Variables
• Device Selection
• Memory Management
• Server Configuration
• Networking and Authentication
• Generation Defaults
• Continuous Batching
• Speculative Decoding
• Chat Templates

Environment Variables¶

These variables are read at startup and apply to both the generate command and the serve command.

Note: Variables prefixed with LLAMA_ARG_ are intentional compatibility aliases. They allow mlxcel to be used as a drop-in replacement for llama-server in scripts and deployment configurations that already set these variables.

Device Selection¶

How Device Selection Works¶

When mlxcel starts, it selects a compute device in this order:

1. If MLXCEL_DEVICE is set to cpu, the CPU is used unconditionally.
2. If MLXCEL_DEVICE is set to gpu or metal, the GPU is used (an error is raised if no GPU is available).
3. Otherwise, the GPU is used automatically if available, and the CPU is used as a fallback.

On Apple Silicon, the GPU is the Metal-accelerated unified memory device. On Linux, it is the first CUDA-capable GPU (cuda:0). There is currently no multi-GPU round-robin within a single process; multi-GPU support is handled through the distributed inference features.

Forcing CPU Mode¶

CPU mode is useful for debugging model correctness or running on machines without compatible GPU hardware:

MLXCEL_DEVICE=cpu mlxcel generate -m ./models/my-model -p "Hello"

CPU inference is significantly slower than GPU inference for most models. It is not recommended for production use.

Memory Management¶

The Wired Memory Concept¶

On Apple Silicon, MLX uses "wired memory", memory that is pinned and cannot be swapped to disk. Wired memory is critical for performance because GPU operations require memory to be resident. However, over-wiring can cause system instability or OOM (out-of-memory) errors if other applications also need memory.

The MLXCEL_WIRED_LIMIT variable controls how much GPU memory mlxcel is allowed to wire.

Accepted Values¶

Choosing a Wired Limit¶

For most users running mlxcel as the primary GPU workload on a dedicated machine, leaving MLXCEL_WIRED_LIMIT unset (the default) is the correct choice: mlxcel sets the wired limit to the GPU maximum, matching the mlx-lm pattern for keeping model weights resident.

If you experience memory pressure from other processes, set an explicit limit:

# Reserve 96 GB for mlxcel on a 128 GB system, leaving 32 GB for the OS and other apps
MLXCEL_WIRED_LIMIT=96GB mlxcel serve -m ./models/my-large-model --port 8080

Important: On Linux (CUDA), MLXCEL_WIRED_LIMIT has no effect. CUDA memory management is handled by the CUDA runtime. Use standard CUDA environment variables (CUDA_VISIBLE_DEVICES, etc.) to control GPU allocation on Linux.

Server Configuration¶

Server configuration is applied via command-line flags when starting mlxcel serve or mlxcel-server. Many flags also read from environment variables (documented in the table above).

Networking and Authentication¶

Example: start the server accessible from the local network, with authentication:

mlxcel serve \
    --model ./models/llama-3.1-8b-4bit \
    --host 0.0.0.0 \
    --port 8080 \
    --api-key my-secret-key

Generation Defaults¶

These flags set the default sampling parameters applied to requests that do not specify their own values. Per-request parameters always take precedence over these defaults.

Continuous Batching¶

Continuous batching allows mlxcel to serve multiple concurrent requests efficiently by grouping their decode steps into a single forward pass. This significantly improves throughput under load without increasing latency for individual requests.

For llama.cpp compatibility, --ctx-size C is interpreted as a total context budget for the server when C > 0. The effective per-slot context window is floor(C / active_slots), where active_slots is --max-batch-size when explicitly set, otherwise --n-parallel; --no-batch uses one slot. /health.context_size, /slots[].context_size, and --estimate-memory use this effective per-slot value. Startup rejects explicit budgets that would leave a slot below 512 tokens.

Example: configure a server for high-concurrency use:

mlxcel serve \
    --model ./models/llama-3.1-8b-4bit \
    --port 8080 \
    --n-parallel 8 \
    --max-batch-size 8 \
    --max-queue-depth 64 \
    --prefill-chunk-size 512

The explicit --prefill-chunk-size 512 in this example pins the chunk at 512 for both idle and contended ticks; omit it to keep the adaptive 2048/512 default. See Continuous Batching for how chunking and the KV admission watermark interact with the scheduler.

Speculative Decoding¶

Speculative decoding accelerates generation by using a drafter to propose tokens that the target model verifies. Server mode supports the DFlash and MTP drafter kinds, an ordinary smaller checkpoint as a full-model drafter, and drafter-free prompt lookup (--draft-kind prompt-lookup).

DFlash example:

mlxcel serve \
    --model ./models/qwen3.5-27b-4bit \
    --draft-model ./models/Qwen3.5-27B-DFlash \
    --draft-kind dflash \
    --draft-block-size 16 \
    --port 8080

Chat Templates¶

mlxcel automatically reads the chat template from the model's tokenizer_config.json. You can override this with a custom template.

Template resolution order:

1. --chat-template (inline string)
2. --chat-template-file (file path)
3. tokenizer_config.json chat_template field (automatic)
4. Built-in fallback template

Optional Endpoints¶

Warmup¶

By default, the server generates one token on startup (and discards it) to pre-compile the Metal or CUDA compute pipeline. This eliminates latency on the first real request.

# Disable warmup (faster startup, but first request will be slower)
mlxcel serve --model ./models/my-model --no-warmup