Qwen

Qwen/Qwen3.8-27B

Qwen3.8 27B multimodal serving with optional Metal MTP speculative decoding.

Start this recipe View on Hugging Face

Model metadata

CategoryVLMParams27BContext262,144BackendsCUDA, METALTasksChat, Vision, Video, Reasoning, Tool calling

Recipe authorship · Recorded configuration verification

Recipe authorship: Curated

Curated and generated describe how this recipe was written, not whether it ran successfully.

Recorded configuration verification

No current matching configuration verification record.

Records cover the tested default configuration on the named hardware. Matching release minor, variant, and rendering hash are required; changing options does not verify the new configuration.

Choose your actual device. Your browser cannot detect GPU memory.

Command builder

Hardware

Variant

mlx-community/Qwen3.8-27B-4bit

Mode: Server

Advanced

Mode

Context

Context length is measured in tokens. Use the slider or enter a number.

Slots

Features

Memory estimation: Why unavailable?

This feature is not available in the selected mode.

Estimated memory use

approx

Approximate memory allocation in GB. These estimates are not measured peak usage.

Weights: ≈ 14.1 GB
KV: ≈ 2.0 GB
Drafter: ≈ 0.0 GB
Runtime allowance: ≈ 1.7 GB

Estimated to leave ≈ 3.6 GB of the ≈ 21.4 GB available memory.

Next: prepare the executable

Notes

This tier is estimated from nearby measurements.

1. Install

Install in your terminal. Uncached model IDs download automatically on first use.

MLXCEL_INSTALL_DIR="${MLXCEL_INSTALL_DIR:-$HOME/.local/share/mlxcel}" \
  && mkdir -p "$MLXCEL_INSTALL_DIR" \
  && curl -fL -o "$MLXCEL_INSTALL_DIR/mlxcel.zip" https://github.com/lablup/mlxcel/releases/download/v0.8.1/mlxcel-macos-aarch64.zip \
  && unzip -o "$MLXCEL_INSTALL_DIR/mlxcel.zip" -d "$MLXCEL_INSTALL_DIR/mlxcel-macos-aarch64" \
  && export PATH="$MLXCEL_INSTALL_DIR/mlxcel-macos-aarch64:$PATH" \
  && mlxcel --version

Installation and download help

Run the install command in your terminal. The selected executable should then be available on PATH. If it is not found, repeat the displayed PATH export in the same terminal.

Model IDs download automatically when first used. The optional download command prepares the checkpoint ahead of time. Uncached models require internet access and enough disk space; gated models also require access approval and credentials.

2. Run

Run this command in the same terminal after preparation. For server modes, keep it running and wait for the listening/ready message. If memory is insufficient, select a smaller checkpoint or reduce context/concurrency in advanced settings.

Copy all includes the selected installation, checkpoint download, and displayed run commands, in that order. Run them in the same shell; multi-node commands belong on their respective machines.

mlxcel serve

This installation provides the full CLI. Use mlxcel serve; select a source build to use the standalone server.

mlxcel serve -m mlx-community/Qwen3.8-27B-4bit --alias qwen3.8-27b \
  --ctx-size 8192 --n-parallel 1

Command flag help

-m: Selects the model or checkpoint repository.
--alias: Sets the served model name used by API clients.
--ctx-size: Sets the maximum context window for requests.
--n-parallel: Sets the number of parallel server slots.

3. Check the first result

For a server, use a second terminal after it is ready. A text chat request returns JSON with choices[0].message.content; specialized endpoints return their task output. Replace sample file paths with your own inputs where shown. Connection refused: check the server address/port and terminal logs. Model access/download errors: check the model ID, permissions, internet connection, and free disk space.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":[{"type":"text","text":"Describe this image"},{"type":"image_url","image_url":{"url":"https://example.com/photo.jpg"}}]}]}'

Performance evidence

No benchmark is recorded for this selected checkpoint and hardware combination. Commands and memory estimates use the catalog's model information.

Memory fit is a model-based estimate, separate from measured throughput and successful inference verification.

No measurement is available for this hardware yet. See the family page for related profiles. Family

Prerequisites

Install the mlxcel build for your selected hardware and check the selected checkpoint's memory estimate.
Allow internet access and enough disk space to download an uncached checkpoint; obtain Hugging Face access approval when the checkpoint requires it.

Recipe options

• MTP is supported on Metal for this family and applies to text token generation. The target model processes image and video inputs.
• Qwen3.8 MTP is single-request only. With more than one parallel slot, mlxcel falls back to classic decode for batched windows.
• The Qwen MTP drafter is approximately 0.425B parameters, derived from the documented 810 MiB bf16 checkpoint size.

Family

Qwen 3.5 / 3.8 VLM

VLM

Sibling recipes

pending

Guide

Qwen3.8 27B

Use this recipe for multimodal Qwen3.8 serving, or add the matching MTP head for lower-latency single-request text decoding on Apple Silicon.

Start with the target

Select an Apple memory tier with enough room for the target, KV cache, and headroom.
Keep Slots at 1 when using MTP.
Enable spec decoding and keep the drafter set to mtp.
Copy the generated download and run commands.

The builder always pairs the 4-bit target with mlx-community/Qwen3.8-27B-MTP-bf16; it does not accept an arbitrary drafter path. The fit meter accounts for both checkpoints.

Operational notes

This Qwen MTP pairing is supported on Metal, so the builder hides it on CUDA.
The target model processes image and video inputs. MTP applies to text token generation.
With more than one parallel slot, the server uses ordinary decoding. Set one slot to use MTP.
DFlash is server-only and no published DFlash checkpoint is paired with this recipe; a CLI DFlash request resolves to MTP.

For static cluster discovery, manual partitions, and deeper tuning, see the mlxcel manual.