LoRA Adapters¶

What mlxcel Implements¶

LoRA represents a weight update with two low-rank matrices:

W_fused = W_base + scale × (B @ A)

mlxcel reads matching .lora_a and .lora_b tensors from an adapter directory and applies them in one of two ways.

• Fused. The delta is added to the base weight map before model construction. mlxcel generate --adapter always fuses, and the server fuses with --lora-fuse and under pipeline parallelism. A fused adapter adds no work to decode, and its scale cannot change after load.
• Unfused runtime. mlxcel serve and mlxcel-server keep each adapter as a low-rank term with a live scale. The scale is set at startup (--lora is scale 1.0, --lora-scaled sets it, --lora-init-without-apply starts at 0.0), changed with POST /lora-adapters, or chosen per request. This is the server default whenever --adapter / --lora is given.

Several adapters can be loaded at once, unfused or fused with --lora-fuse, and their terms sum.

Directory Contract¶

An adapter directory must contain:

adapter_config.json
adapters.safetensors          # preferred

If adapters.safetensors is absent, mlxcel tries the single fallback filename adapter_model.safetensors. Sharded adapter files and arbitrary collections of .safetensors files are not discovered.

The loader recognizes only tensor keys ending exactly in the following, and refuses any other tensor in the file (a DoRA magnitude vector, a stray .weight):

<base-weight-stem>.lora_a
<base-weight-stem>.lora_b

Both matrices must be present for a pair. The stem is resolved to the corresponding base weight and its final delta shape must match exactly; mlxcel refuses unsafe transposed/broadcastable mismatches. It accepts the mlx-lm orientation (in, rank) / (rank, out) and the common matrix orientation (rank, in) / (out, rank) when dimensions are unambiguous.

This contract is not blanket HuggingFace PEFT compatibility. A file named adapter_model.safetensors is accepted, but PEFT key names such as .lora_A.weight, sharded indexes, or a differently shaped configuration must be converted to mlxcel's key/config contract.

Configuration¶

adapter_config.json uses mlxcel/mlx-lm-style nested parameters:

{
  "fine_tune_type": "lora",
  "lora_parameters": {
    "rank": 16,
    "lora_alpha": 32,
    "dropout": 0.0
  },
  "num_layers": -1,
  "target_modules": ["q_proj", "k_proj", "v_proj", "o_proj"],
  "model": "mistralai/Mistral-7B-v0.3"
}

fine_tune_type: "dora" is refused at load with DoRA adapters are not supported, because applying only the low-rank pair would produce weights that match neither the base model nor the fine-tune. Fuse a DoRA adapter into its base checkpoint with its own tooling first.

Other refusals, all raised at load before any weight is changed: a pair with no matching base weight; a 3-D pair (a mixture-of-experts expert adapter, mlx-lm LoRASwitchLinear); PEFT .lora_A.weight / .lora_B.weight naming; an adapter that applies no tensors; and fusing into a quantized base weight (the error points to the unfused --lora path, which serves adapters on quantized bases).

Embedding and Output-Head Pairs¶

An embed_tokens pair in the mlx-lm LoRAEmbedding layout (lora_a [vocab, rank], lora_b [rank, hidden]) is applied on the input lookup and, when the output head is tied to the embedding, on the head as well. Two linear-shaped orientations of the same projection are also accepted, matched against the vocabulary size. Fused (--adapter on generate, --lora-fuse) and unfused (--lora) loading both support it, but only for the Llama decoder (llama, mistral, qwen2) with tie_word_embeddings: true and for Qwen3 unless its config sets tie_word_embeddings: false. On every other family an embed_tokens pair on a tied checkpoint is refused, and so is an lm_head pair on any tied checkpoint, on every load path.

CLI¶

mlxcel generate \
  --model ./models/Mistral-7B-v0.3 \
  --adapter ./adapters/mistral-sql-lora \
  --prompt "Write a SQL query for orders over 1000 dollars." \
  --max-tokens 200

mlxcel generate --adapter fuses the adapter at load. Unsupported model families fail explicitly instead of silently ignoring the adapter.

Server¶

mlxcel serve \
  --model ./models/Mistral-7B-v0.3 \
  --adapter ./adapters/mistral-sql-lora \
  --port 8080

--adapter and --lora are the same flag on the server. It takes a comma-separated list of adapter directories at scale 1.0. --lora-scaled FNAME:SCALE,... adds adapters with explicit scales (the entry is split on the last colon, and a non-numeric or non-finite scale is a startup error). --lora-init-without-apply loads every listed adapter at scale 0.0. --lora-fuse bakes the adapters into the weights instead of keeping them unfused. See the CLI reference for the flag table.

Adapters stack by summing their terms, so the order does not change the result. The prompt-cache key includes a digest of the effective adapter scales on the unfused path, so cached prefixes computed under other scales are never reused.

Tensor parallelism does not support LoRA (the runtime refuses it at startup). Pipeline parallelism accepts one unscaled adapter, fused, and --lora-scaled or several --lora entries with tensor or pipeline parallelism are a startup error. In-process pipeline parallelism filters the adapter tensors to each stage. Remote pipeline startup rejects adapters. Muse Glimmer rejects adapters. VLM and special-model support varies by loader; rely on the explicit startup error and validate the intended checkpoint.

Performance and Memory¶

A fused adapter removes the separate adapter matrix multiplication from each generated token, so there is no LoRA branch in the decode loop. An unfused adapter adds its low-rank products to every forward pass in exchange for runtime scale control and quantized-base support. Fusion does not mean "zero cost" in every dimension:

• Startup performs matrix products and additions for every adapter pair.
• The ordinary non-pipeline path clones the base weight map before fusion, which can increase peak load memory; the stage-local pipeline path fuses its stage map in place.
• The resulting fused tensors still occupy memory, and quantized/packed base layouts are not dequantized and requantized to make an adapter fit; fusing into one is refused. Fused weights keep the base dtype.
• Actual throughput can vary with dtype, weight layout, backend, and whether fusion changed the model's load representation.

Always run a short output comparison and a representative memory/throughput test before deployment.