Performance Tuning¶

mlxcel chooses conservative launch and attention policies automatically. Tune only after measuring a representative prompt length, batch size, model, cache mode, and device. A value measured on one GPU or context shape is not a portable default.

Kernel Autotuning¶

mlxcel tune profiles supported launch candidates and stores winners under ${MLXCEL_CACHE_DIR:-$HOME/.cache/mlxcel}/autotune. Writing the cache does not change inference until MLXCEL_AUTOTUNE is enabled.

# Print the matrix without profiling
mlxcel tune --dry-run

# Tune all operations supported by this backend
mlxcel tune

# Tune paged decode using a checkpoint's head geometry
mlxcel tune --op paged-decode-splits -m ./models/qwen3-4b-4bit

# Consume only cached winners
MLXCEL_AUTOTUNE=cache mlxcel serve -m ./models/qwen3-4b-4bit

# Consume cached winners and tune unseen shapes on first use
MLXCEL_AUTOTUNE=1 mlxcel serve -m ./models/qwen3-4b-4bit

Supported operations are paged-decode-splits on Metal and CUDA, plus currently unvalidated CUDA-only qmm-tile and qmv-multirow sweeps. The default matrix covers batch sizes 1,2,4,8 and context lengths 1024,4096,16384. Use --model or explicit --head-dim, --q-heads, --kv-heads, and --block-size to match production geometry. The profiler uses warmup, adaptive sampling, median scoring, and a default 2% minimum improvement so noise does not replace a stable default.

Attention Dispatch¶

The server's --decode-storage-backend auto default selects paged decode only for qualified model families and falls back to dense storage elsewhere. Leave it on auto unless a controlled comparison proves a different choice on the exact workload. The current automatic paged set includes Llama, Qwen 3, Qwen 3.5 text, Gemma 3 text, and Llama 4 workers; internal-cache and hybrid families remain dense until they provide model-owned paged dispatch.

Advanced paths are also policy-gated:

The primary sparse-paged controls are MLXCEL_SPARSE_PAGED_ATTENTION and MLXCEL_SPARSE_PAGED_MIN_SPARSITY. Sparse paged decode is token-exact for the implemented MiniMax-M3 route. DeepSeek Sparse Attention is not routed through that kernel because its positional score requires an additional K stream that the current v2 kernel does not accept. CUDA implementations of several advanced kernels remain unvalidated; do not infer CUDA qualification from a Metal result.

Host and Command-Buffer Controls¶

MLX_MAX_OPS_PER_BUFFER automatically defaults to 1000 on pre-M5 Apple Silicon (M1-M4) and keeps the MLX default on M5 or non-Apple targets. On the same pre-M5 hosts, MLXCEL_DECODE_MB_PER_BUFFER raises the Metal command-buffer input budget to 1000 for pipelined decode steps only: with the op cap raised, the input budget is the limit that binds, and at MLX's default a decode token commits a buffer every one or two layers. Prefill keeps the device default because the larger budget roughly doubles peak memory on long prompts. Setting MLX_MAX_MB_PER_BUFFER yourself pins one value for both phases and turns the decode-only raise off.

On CUDA, MLX_CUDA_GRAPH_CACHE_SIZE and MLX_CUDA_SDPA_CACHE_SIZE default to 2000 to avoid cache thrashing in long-lived, shape-diverse serving. On compute capability 12.1 (GB10), MLX's graph capture budgets are the tightest in its table, and mlxcel raises them (MLX_MAX_OPS_PER_BUFFER=100, MLX_MAX_MB_PER_BUFFER=1000) only for model families that gained on every measured workload, which is laguna today; other families gained on some workloads and lost on others. mlxcel generate prints the applied budget next to its compute-capability line and mlxcel-server logs it. An explicit environment value always wins.

Model load reads and uploads a generation model's checkpoint tensors before it reports Model loaded, so the first token pays no weight I/O and the reported load time covers resident weights. Tensors of vision and audio towers stay lazy until a request carries that modality. MLXCEL_WEIGHT_MATERIALIZATION=lazy materializes each tensor on first use instead. On CUDA, load stages tensors through four reused 64 MiB pinned host buffers; MLXCEL_CUDA_PINNED_LOAD=0 forces the pageable path. MLXCEL_PROFILE_TTFT adds the upload counters to the [TTFT] line.

On M1 and M2, dense products of 2 to 6 rows run on the GEMV kernel, which raises batched decode at small batch sizes. MLXCEL_DENSE_GEMV_ROWS=0 restores the upstream dense dispatch. On CUDA, MLXCEL_SDPA_DETERMINISTIC=1 selects a reproducible decode SDPA path for comparisons that need identical output between runs.

Decode runs on one batch-native engine: a single sequence is a batch of one, and mlxcel run, mlxcel generate, and the server share the same sampling and finish step. Most model families decode natively as a batch, and the server logs decode path: native batched forward per model at startup.

On M1 Ultra, batched decode still has measured gaps: B=1 decode is 3 to 6% lower than before the batched decode work on a few checkpoints, and Mixtral stays below its B=1 rate at B=8 with a warm prompt cache (#2817). Measure your own checkpoint at the batch size you serve before raising --max-batch-size.

For server throughput, tune --n-parallel, --max-batch-size, --prefill-chunk-size, --prefill-grant-interval, --batch-size, and --ubatch-size together. Raising concurrency increases throughput only while the model, KV cache, and temporary tensors fit. Use Memory Estimation, /metrics, and a steady-state load test instead of relying on load time alone.

Measurement Checklist¶

1. Record the exact mlxcel version, checkpoint revision, device, cache mode, prompt length distribution, output length, and concurrency.
2. Warm the model and kernel cache before collecting samples.
3. Compare prefill tokens/s, decode tokens/s, time to first token, tail latency, memory high-water mark, and request failures.
4. Run one change at a time and repeat enough samples to overcome thermal and background-load noise.
5. Keep a kill switch or the auto baseline available. If an advanced path reports a fallback, do not claim its performance from that run.