Vision-Language Models¶

Table of Contents¶

• What Are Vision-Language Models?
• The VLM Pipeline
• Supported VLM Families
• Florence-2
• Falcon-OCR layout-aware OCR
• GOT-OCR 2.0
• Nemotron-Parse
• Vision Encoders
• Vision-Text Connectors
• Using VLMs from the CLI
• Using VLMs from the API
• Video and Audio Inputs
• Video on a model with no native video path
• Vision Feature Cache

What Are Vision-Language Models?¶

A vision-language model (VLM) is a model that can understand both images and text. At inference time, one or more images are passed alongside a text prompt, and the model produces a text response that takes both modalities into account.

VLMs are used for tasks such as:

• Image description - Generating captions or detailed descriptions of photographs, diagrams, or screenshots.
• Visual question answering - Answering questions about the content of an image.
• Document understanding - Extracting structured information from scanned documents, charts, or tables.
• Code and UI assistance - Analyzing screenshots of code editors, web pages, or application interfaces.
• Multi-image comparison - Describing differences or relationships between two or more images.

mlxcel treats a VLM as a text language model with a vision module attached. The text model is loaded exactly as a standard language model; the vision module is an additional set of weights that processes images into the same embedding space as text tokens.

The VLM Pipeline¶

When an image is passed with a prompt, mlxcel processes it through a four-stage pipeline before the combined representation reaches the text model.

flowchart LR
    IMG[Raw Image] --> PROC[Image Processor\nresize / normalize / tile]
    PROC --> ENC[Vision Encoder\nViT / SigLIP / Pixtral]
    ENC --> CONN[Connector\nMLP / Linear / Merger]
    CONN --> MERGE[Embedding Merge\ninsert at image token positions]
    TEXT[Text Tokens] --> EMBED[Text Embedding Table]
    EMBED --> MERGE
    MERGE --> LLM[Text Language Model\nforward pass]
    LLM --> OUT[Generated Tokens]

Stage 1: Image Processor

The image processor handles the raw input image. Depending on the model family, this stage may: - Resize the image to a fixed resolution. - Tile the image into a grid of fixed-size patches for high-resolution support. - Normalize pixel values to the range and mean/variance expected by the encoder.

Stage 2: Vision Encoder

The vision encoder is a separate neural network (typically a Vision Transformer, or ViT) that converts the processed pixel data into a sequence of patch-level hidden state vectors. Each patch corresponds to a region of the image. The output is a tensor of shape [batch, num_patches, hidden_dim].

Stage 3: Connector (Projector)

The connector is a small projection network that maps the vision encoder's hidden dimension to the text model's hidden dimension. This is necessary because the two networks often have different widths. The simplest connector is a single linear layer; more complex connectors use multi-layer MLPs or spatial pooling.

Stage 4: Embedding Merge

The projected image features are inserted into the text token embedding sequence at the positions occupied by special image placeholder tokens. The resulting combined sequence is passed to the text language model as if it were a standard sequence of token embeddings. The text model is unmodified.

Supported VLM Families¶

mlxcel includes loaders for the following VLM families, identified automatically from the model_type field in config.json:

Text-only checkpoints of a VLM family¶

The Qwen-VL families (Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3.5-VL) and Mage-VL also load vision-stripped checkpoints. A checkpoint loads without a vision tower when config.json has no vision_config, vision_config is null or {}, top-level language_model_only is true, or the safetensors index carries no vision weights (vision_tower.*, model.visual.*, and the other tower prefixes). Text requests then run normally, and an image or video request fails with model <path> was loaded without a vision tower (text-only checkpoint). language_model_only: true wins even when vision weights ship, and mlxcel prints a stderr notice saying the checkpoint loads text-only; for Qwen 3.5 a non-boolean language_model_only value is rejected at load.

Florence-2¶

Florence-2 is an encoder-decoder VLM (like Nemotron-Parse), and it is driven by a task marker rather than a free-form instruction. The DaViT tower encodes the 768x768 image into a feature grid, the projected image tokens are concatenated in front of the task-prompt embeddings, and a BART decoder generates the answer greedily with cross-attention against the cached encoder output.

The -p/--prompt string selects one of fifteen task modes. Markers are case-insensitive and the angle brackets are optional:

mlxcel generate -m models/Florence-2-base-ft-bf16 --image page.png -p '<OD>'
mlxcel generate -m models/Florence-2-base-ft-bf16 --image photo.jpg -p '<DETAILED_CAPTION>'
mlxcel generate -m models/Florence-2-base-ft-bf16 --image doc.png -p '<OCR_WITH_REGION>'

Seven of the task modes take input text after the marker, for example <CAPTION_TO_PHRASE_GROUNDING> a red car. Spatial answers come back through the checkpoint's 1000-bin <loc_*> tokens and are parsed into pixel-space boxes, quad boxes, or polygons against the original image size.

Over HTTP, send an OpenAI-compatible POST /v1/chat/completions with one user message carrying the task prompt as its text and one image_url content part. message.content holds the same rendered text the CLI prints, and the parsed coordinates arrive as JSON in the mlxcel-specific message.florence2_result extension field.

Serving is one request at a time by design. The encoder pass has a different cost profile from the decode loop, so requests queue on a dedicated worker channel (bounded by --max-queue-depth) and are answered serially rather than through the batched or paged scheduler. Decode is greedy, so sampling parameters and response_format are accepted and ignored, and a stream: true request delivers the whole answer as a single delta chunk because the parsed result only exists after the full decode.

Dense and quantized exports both load. The -large-ft releases load but do not generate usable text on the published MLX conversions; prefer -base-ft.

Falcon-OCR layout-aware OCR¶

Falcon-OCR transcribes a whole page in one pass by default. Passing a layout detection file switches it into a layout-aware second stage that routes by category, suppresses nested boxes, crops regions, and applies per-category instructions:

mlxcel generate -m models/falcon-ocr -i page.png --layout-detections layout.json

Detection stays a separate command: the detections are an input to generate. mlxcel detect --format json with a PP-DocLayoutV3 checkpoint (the reference layout detector, which emits boxes in reading order) or an RT-DETRv2 layout checkpoint writes the file. generate keeps the order of the file it is given, and mlxcel detect sorts RT-DETRv2 boxes top to bottom and left to right, because that model has no reading-order head.

GOT-OCR 2.0¶

GOT-OCR 2.0 (model_type: GOT) is a 0.58B document-OCR model. Every page is stretched onto a fixed 1024x1024 canvas and becomes exactly 256 feature rows, so there is no tiling and no aspect-ratio preservation. One image per request is accepted.

The checkpoint ships no chat template. mlxcel supplies the fixed conversation the reference implementation builds, on the CLI and on the server alike, and replaces any client system message with that fixed one. The instruction text selects the mode:

mlxcel generate -m models/GOT-OCR2_0-bf16 --image page.png -p "OCR: "

Generation stops on the <|im_end|> turn separator as well as <|endoftext|>. Multi-crop OCR is not implemented.

Nemotron-Parse¶

Nemotron-Parse (model_type: nemotron_parse) is NVIDIA's document OCR and layout model, an encoder-decoder VLM. The page is shrunk to fit 2048x1664 with its aspect ratio kept and padded with white to that size. A C-RADIOv2-H tower encodes it into 3329 encoder states, and a 10-layer mBART decoder, seeded with the tokenized task prompt, decodes greedily until </s>. The answer is markdown annotated with <x_..><y_..> box tokens (page-normalized) and <class_..> tags, printed with its special tokens kept.

Like Florence-2, it runs on its own CLI driver and a dedicated single-stream server worker rather than the autoregressive loop or the batched scheduler. The task prompt is the -p string on the CLI:

mlxcel generate -m models/Nemotron-Parse-2.0-8bit --image page.png \
  -p '</s><s><predict_bbox><predict_classes><output_markdown><predict_no_text_in_pic>' -n 4096

The trained prompts combine these control tokens after </s><s>:

An empty prompt makes the model loop, so both front ends substitute the default prompt (the CLI prints a warning). --repetition-penalty and the request's repetition_penalty follow transformers semantics over the whole decoder sequence, seed included; the model card's script uses 1.1, and mlxcel's default stays 1.0.

Over HTTP, send POST /v1/chat/completions with one user message carrying the task prompt as its text part and one image_url part. The server bounds the task prompt (1024 bytes, no control characters, at most 256 seed tokens), requires exactly one image, refuses audio and video, and answers a streaming request with one delta. Batch sizes above one and the model card's table-insertion and repetition-stop logits processors are not supported.

Vision Encoders¶

mlxcel implements several vision encoder architectures. The correct encoder is selected automatically based on the model's configuration.

SigLIP

Used by: PaliGemma, Aya Vision, Phi 4 SigLIP, and some Gemma 3 variants. SigLIP (Sigmoid Loss for Language-Image Pre-training) is a ViT trained with a binary cross-entropy loss rather than the contrastive loss used by CLIP. The encoder produces patch embeddings at a fixed resolution.

Llama 4 Vision Encoder

Used by: Llama 4 models. A native vision transformer designed for Llama 4's multi-modal architecture.

Qwen2-VL ViT with MRoPE

Used by: Qwen2-VL, Qwen2.5-VL. This encoder extends standard ViT position encodings with Multimodal Rotary Position Embedding (MRoPE), which encodes both the spatial position within an image and the temporal position of the image in a multi-turn conversation.

Pixtral 2D RoPE ViT

Used by: Pixtral, Mistral 3. A Vision Transformer with 2D RoPE position encodings that preserve the two-dimensional structure of the image grid.

CLIP Variants

Used by: Phi 3 Vision and related models. Standard CLIP ViT encoders with contrastive pre-training.

Moondream3 Encoder

A dedicated encoder for the Moondream3 architecture with custom image preprocessing.

Molmo2 Encoder

A dedicated encoder for the Molmo2 architecture.

MiniCPM-o Encoder

A dedicated encoder for the MiniCPM-o architecture.

Vision-Text Connectors¶

After the vision encoder produces patch embeddings, a connector maps them into the text model's hidden space. The connector is always a small network relative to the vision encoder and the language model.

Using VLMs from the CLI¶

The mlxcel generate command accepts one or more image paths via --image. The model must support vision input; passing images to a text-only model results in the images being ignored.

Single image:

mlxcel generate \
  --model ./models/Qwen2.5-VL-7B-Instruct \
  --image ./photo.jpg \
  --prompt "What is in this image?" \
  --max-tokens 200

Multiple images:

mlxcel generate \
  --model ./models/Qwen2.5-VL-7B-Instruct \
  --image ./before.png --image ./after.png \
  --prompt "What are the differences between these two screenshots?" \
  --max-tokens 300

Document understanding example with PaliGemma:

mlxcel generate \
  --model ./models/PaliGemma-3B \
  --image ./invoice.png \
  --prompt "Extract the total amount from this invoice." \
  --max-tokens 50

Chat template considerations:

For most VLMs the chat template automatically inserts the image placeholder token at the correct position relative to the prompt text. If you need precise control over image placement within a multi-turn conversation, use --no-chat-template and format the prompt according to the model's expected template format manually.

Using VLMs from the API¶

When using the /v1/chat/completions endpoint, images are passed as part of the message content using the OpenAI multimodal message format. Images can be provided as a URL, a local path, or as a Base64-encoded data URI.

Image from URL:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen2.5-VL-7B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "image_url",
            "image_url": {"url": "https://example.com/photo.jpg"}
          },
          {
            "type": "text",
            "text": "Describe this image in detail."
          }
        ]
      }
    ],
    "max_tokens": 300
  }'

Image from Base64 data URI:

IMAGE_B64=$(base64 -i ./photo.jpg)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"Qwen2.5-VL-7B-Instruct\",
    \"messages\": [
      {
        \"role\": \"user\",
        \"content\": [
          {
            \"type\": \"image_url\",
            \"image_url\": {\"url\": \"data:image/jpeg;base64,${IMAGE_B64}\"}
          },
          {
            \"type\": \"text\",
            \"text\": \"What text appears in this image?\"
          }
        ]
      }
    ],
    \"max_tokens\": 200
  }"

Python example with the OpenAI SDK:

import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="unused")

with open("./diagram.png", "rb") as f:
    image_data = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="Qwen2.5-VL-7B-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image_data}"},
                },
                {
                    "type": "text",
                    "text": "Explain what this architecture diagram shows.",
                },
            ],
        }
    ],
    max_tokens=400,
)
print(response.choices[0].message.content)

Video and Audio Inputs¶

mlxcel generate also accepts video and audio media on the same prompt surface:

mlxcel generate \
  --model ./models/gemma-4-vl \
  --video ./clip.mp4 \
  --fps 2.0 \
  --prompt "Describe the clip." \
  --max-tokens 200

mlxcel generate \
  --model ./models/gemma-4-audio \
  --audio ./question.wav \
  --prompt "Transcribe and answer the question." \
  --max-tokens 200

Frame extraction requires ffmpeg on PATH; --fps controls the target sampling rate and defaults to 2.0. Families with a native video path consume the clip through their own temporal processor: Gemma 4 VL, Gemma 4 Unified, Inkling, Kimi-VL / Kimi-VL 2.5, and the Qwen-VL families (Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3-VL MoE, Qwen3.5-VL and its MoE variant, and Qwen3.8-Flash-Next). Every other checkpoint with a vision tower takes the frames fallback described below.

The server accepts video and audio content parts in chat messages:

{
  "role": "user",
  "content": [
    {"type": "video_url", "video_url": {"url": "file:///srv/videos/demo.mp4", "fps": 2.0}},
    {"type": "text", "text": "Summarize the visible events."}
  ]
}

For local video_url paths, set MLXCEL_VIDEO_DIR_ALLOWLIST to a comma-separated list of directories the server may read. The default is fail-closed: local video paths are rejected unless an allowlisted directory contains them. Both native-video families and the frames fallback resolve the list once when the server starts (once per model the router loads), so restart the server after changing the variable or creating a listed directory; request handling does not re-read it. Data-video URIs and HTTP(S) URLs are written as create-new files with mode 0600 inside private 0700 directories before ffmpeg reads them. When the loaded checkpoint takes video, the server checks for ffmpeg and ffprobe once at startup, logs a warning if either is missing, and keeps that answer for the life of the process, so install ffmpeg before starting the server.

Audio content uses the OpenAI input_audio shape. The currently wired audio route expects WAV payloads:

{
  "type": "input_audio",
  "input_audio": {
    "data": "data:audio/wav;base64,...",
    "format": "wav"
  }
}

Video and audio in the same prompt are accepted only by Gemma 4 Unified: --video together with --audio on the CLI, or a video_url block together with an input_audio block on the server. The frame runs and the audio run reach the backbone together, and the one-audio-clip-per-request limit still applies. Every other family refuses the combination; the server answers Combined video and audio inputs are not supported with a 400 error. Image+video and image+audio requests are supported where the loaded model implements those paths.

Video on a model with no native video path¶

A checkpoint with a vision tower and no native video path receives a clip as ordered still images. The clip is decoded at the requested fps (video_url.fps, else --video-fps on the server or --fps on the CLI), evenly subsampled to at most --video-max-frames frames (default 16, minimum 2) with the first and last frame always kept, PNG-encoded, and sent as that many images in chronological order. The frames are preceded by the sentence Here is a video as a sequence of N frames in chronological order., so the model reads them as one clip; several clips get one sentence each, placed ahead of that clip's frames. The CLI renders its --video clips after any --image inputs and before the question, which produces the same prompt as a server body that lists the same clips ahead of its question.

The rest of the image pipeline is unchanged: the template emits one image placeholder per frame, and the prompt-cache multimodal digest hashes the frame bytes, so two requests for the same clip at the same fps and frame cap share a prefix. On the CLI the frames are written into a private per-run temporary directory that is removed when the run ends.

What the fallback costs and what it does not provide:

• The prompt grows by the model's per-image token cost times the kept frame count; at 256 tokens per image, the 16-frame default adds about 4,096 prompt tokens.
• Frames count against the per-request image budget (--max-images, default 16). A request that goes over it is refused before decoding, and the refusal names the frame count, the caller's own image count, and the limit.
• There is no temporal encoding. The model sees an ordered set of pictures and the sentence saying they are one video, with no timestamps and no motion between frames.

Muse Glimmer refuses --video and video_url on both front ends. A checkpoint with no vision tower refuses video. GET /props reports video: true for both native and fallback families; the server log line model <id> has no native video path; sending K of N sampled frames from <file> as ordered images shows that the fallback was used.

Vision Feature Cache¶

Multi-turn VLM conversations that revisit the same image pay the full vision-tower cost on every turn by default: the image is preprocessed, run through the vision encoder, projected through the multimodal connector, and only then used to seed the language model. For encoders like SigLIP or Qwen's ViT this is the dominant non-prefill cost on every later turn, even when the image has not changed.

mlxcel-server keeps an LRU of post-projection image features per loaded VLM so that a cache hit short-circuits both the vision tower and the multimodal embedder. The cached entry is the same tensor the language model would have received, so reuse is bit-exact rather than a lossy approximation.

Supported models¶

The feature cache is wired into:

• Gemma 4 VLM (single-array features via SingleArrayFeatures)
• Qwen 2.5-VL (single-array features)
• Qwen 3-VL (deep-stack features via DeepStackFeatures to carry the multi-level vision pyramid)
• Cohere Compass (deep-stack features)
• LFM2-VL (single-array features, cached per image)

Other VLM families (Qwen 2-VL, Qwen 3.5-VL, Qwen 3-VL-MoE, Molmo / Molmo-Point, Phi 3 Vision, Phi 4 MM, MiniCPM-O, MiniCPM-V 4.6, Moondream 3, Gemma 3 VLM, LLaVA, Llama 4 VLM, Pixtral) fall back to the uncached path automatically via the default trait implementation. Adding another family is a matter of implementing CloneableFeatures for its vision-feature shape and threading _with_cache variants into get_input_embeddings. No request-level changes are required.

Cache keys¶

Keys are constructed from whatever identifies the image at the request boundary:

• Filesystem path when the request specifies a local file or a file:// URL. The path is normalized before being hashed into the key so that ./img.png and /abs/path/img.png resolving to the same file do not duplicate entries.
• SHA-256 digest of the inline payload when the image arrives as a Base64 data URI or an http(s):// download. The hash is computed over the raw compressed bytes (image_hash_from_bytes) so that decoded pixel tensors never have to be hashed.

Path keys and hash keys live in disjoint namespaces in the same cache, so a request that loads /abs/img.png from disk and a subsequent request that uploads the same bytes inline each get their own cache entry. This is intentional: identifying the same image across both delivery modes would require hashing disk reads on every path-based request, which defeats the purpose.

Tuning --vision-cache-size¶

mlxcel-server \
  --model ./models/Qwen2.5-VL-7B-Instruct \
  --vision-cache-size 40 \
  --port 8080

Sizing guidance:

• Chat-style workloads with ~5 distinct images per session and many concurrent sessions: set to max_batch_size * average_images_per_session plus a small margin. The default 20 fits roughly 4 concurrent chat sessions carrying 5 images each.
• Document-understanding workloads where the same invoice/screenshot appears across many turns: a size equal to the number of distinct source documents is usually enough. Values above ~100 rarely help because the hit rate plateaus well before then.
• Memory footprint per entry scales with the vision encoder's output dim times the number of image tokens emitted, not the input image resolution. For Qwen 2.5-VL at image_size=448, a single cached entry is on the order of a few hundred KB; for Qwen 3-VL's deep-stack features it is a few MB per image.

Verifying cache correctness¶

Running the same prompt twice with and without the cache must produce identical output:

# Disable cache
mlxcel-server --model ./models/Qwen2.5-VL-7B-Instruct --vision-cache-size 0

# Enable cache (default 20)
mlxcel-server --model ./models/Qwen2.5-VL-7B-Instruct

Send the same multi-turn request (two turns with the same image) to each server. The generated text should match exactly at temperature=0. qwen2.5-vl-3b-4bit is the reference smoke-test target; a mismatch indicates a regression in the family's _with_cache wiring rather than a cache-sizing issue.

Cache scope and lifetime¶

The cache is owned by BatchScheduler as an Rc<ModelVisionCaches> per loaded VLM, so it is scoped to the lifetime of the loaded model. Reloading the model (different --model path or a server restart) starts with an empty cache. There is no cross-process or on-disk persistence: this is a request-path accelerator for a running server, not a build artifact store.

The legacy no-batch worker (--no-batch) retains the default cache size and does not expose --vision-cache-size; users who need to tune the cache should stay on the default batched worker.