Embeddings and Reranking¶

Generation answers a prompt. Retrieval decides which information the generator should see. A production retrieval-augmented system normally divides that decision into two model-serving stages:

1. an embedding model maps queries and documents into vectors so a large corpus can be searched cheaply;
2. a reranker spends more computation on a small candidate list and produces a more accurate relevance order.

The stages are complementary. An embedding index provides scale and recall; a reranker provides precision. mlxcel supports both stages through offline commands, dedicated server workers, dense and multi-vector embeddings, text and image inputs on supported families, and cross-encoder or generative rerankers.

The Retrieve-Then-Rerank Pipeline¶

flowchart TB
    subgraph Offline indexing
        A[Documents] --> B[Chunk and prefix]
        B --> C[Document encoder]
        C --> D[(Vector index)]
    end
    subgraph Online request
        Q[Query] --> E[Query encoder]
        E --> F[ANN or MaxSim search]
        D --> F --> G[Top-K candidates]
        Q --> H[Reranker]
        G --> H --> I[Top-N evidence]
        I --> J[Generator or application]
    end

Offline, each document or passage is chunked, formatted for the checkpoint, encoded, and written to an index with its text, stable ID, metadata, and encoder version. Online, the query encoder produces one vector or a matrix of token vectors. Approximate nearest-neighbor (ANN) or late-interaction search retrieves the top K candidates. A reranker then scores each query-document pair and the application keeps the top N, normally with N < K.

Retrieval quality has a hard ceiling: a reranker cannot recover a relevant document that is absent from the first-stage candidate set. Tune the embedding stage for Recall@K and the reranker for the final ordering.

What mlxcel Serves¶

mlxcel separates generation, embedding, and reranking as model kinds. An embedding or one-label sequence-classification checkpoint passed to -m is detected before the ordinary generation loader. The generation commands refuse an embedding checkpoint and point to mlxcel embed or /v1/embeddings rather than silently treating its hidden states as logits.

The full wire schemas and error codes are in Embedding and Reranking APIs.

Dense Embeddings¶

A dense bi-encoder processes a query and document independently:

e_q = f_q(q)    e_d = f_d(d)

Because e_d is independent of the current query, document vectors can be computed once. Search compares e_q with stored vectors using dot product, cosine similarity, or distance and an ANN structure.

Pooling¶

Many encoders output [batch, tokens, hidden]. Pooling turns each sequence into one vector.

The resolution order is the explicit --pooling override, checkpoint 1_Pooling/config.json, then the family default. MLXCEL_EMBEDDING_POOLING remains a diagnostic override when the CLI flag is absent. The llama-server spelling --pooling last maps to lasttoken; rank selects reranking mode rather than a pooling kernel; none is refused because the pooled embedding families cannot manufacture token vectors after their forward pass.

Do not select a pooling method by intuition. A checkpoint can return plausible vectors under the wrong pooling rule while its retrieval quality collapses.

Normalization and Matryoshka Dimensions¶

The usual similarity functions are:

dot(q, d)    = e_q dot e_d
cosine(q, d) = (e_q dot e_d) / (norm(e_q) * norm(e_d))

When both vectors are L2-normalized, cosine and dot-product ranking are equivalent. mlxcel uses the checkpoint normalization contract by default and supports the llama-server-compatible --embd-normalize values:

embd_normalize can also be supplied per request. A zero vector remains zero rather than becoming NaN.

The OpenAI dimensions field and mlxcel embed --dimensions keep the first requested components and normalize again when normalization is active. This is useful only when the model was trained for Matryoshka truncation; an arbitrary width may preserve the JSON shape while degrading retrieval.

Length, Padding, and Batching¶

The effective maximum length is the smallest applicable limit from sentence_bert_config.json, tokenizer_config.json, an absolute position table, the optional CLI cap, and mlxcel's hard ceiling of 8192. Text is truncated on the right after special-token insertion while preserving a trailing special token such as [SEP]. Supplied token IDs are truncated verbatim.

Text inputs are sorted by token length, divided into micro-batches, right-padded only to the local batch width, and restored to request order. --embedding-batch-size defaults to 16. Supported image inputs are processed one at a time because their visual geometry differs. With bf16 checkpoints, accumulation geometry can make a vector embedded alone and in a larger batch agree at approximately 0.9999 cosine rather than bit-for-bit; compare retrieval tolerance, not raw floating-point identity.

Dense, Multimodal, and Multi-Vector Families¶

Encoder and Decoder-Backbone Embedders¶

Multimodal Embedders¶

An image is accepted only by a family whose supports_images() contract is true. image_url.url may be a data URI, file:// URL, HTTP(S) URL, or local path and is subject to the shared image limits. The service creates vectors; it does not provide a vector database, metadata filters, or an image-search index.

Late Interaction¶

ColIdefics3 and ColQwen2.5 return one normalized vector per real token, normally 128 dimensions, instead of one sequence vector. The float response shape is [tokens, dimensions]; base64 adds an explicit shape field.

mlxcel compares these matrices with MaxSim:

MaxSim(q, d) = sum_i max_j dot(q_i, d_j)

For each query token, it finds the best matching document token and sums those matches. This is more expressive than a single-vector cosine and cheaper than a full cross-encoder, but it requires a multi-vector-aware index. The score is asymmetric and grows with query token count, so it is not directly comparable across unrelated queries.

Prompt and Prefix Contracts¶

mlxcel generally does not invent task prefixes. The bytes used while embedding the corpus must match those used during model training.

A prefix, instruction, tokenizer revision, pooling mode, normalization mode, output dimension, and truncation limit are all part of the index version. If one changes, re-embed the corpus rather than mixing vectors that merely happen to have the same length.

Rerankers¶

A reranker computes a relevance score for each query-document pair:

r_i = g(q, d_i)

Joint attention is more expressive than comparing independent dense vectors, but the document computation cannot be fully precomputed. Reranking K candidates requires approximately K pair evaluations, so this stage belongs after inexpensive retrieval.

Supported Scoring Types¶

Cross-encoders are identifiable from the checkpoint and can be served directly with -m. A generative reranker looks like an ordinary causal or VLM checkpoint, so --reranker-model is required to select the ranking loader. A classifier with more than one output label is rejected rather than assigning an ambiguous score.

Sequence classifiers use the checkpoint pair template and longest-first truncation, preserving its special tokens. Qwen3 rerankers preserve the system and assistant scoring scaffold while truncating the query/document content. The default instruction describes web passage retrieval and can be overridden. Qwen3-VL renders the checkpoint's reranker.jinja, permits text and/or image objects, and scores image-bearing rows separately; video reranking is not supported.

Scores lie in [0, 1], but they are primarily meaningful within one query's candidate set. mlxcel sorts descending, breaks exact ties by original index, and then applies top_n.

Running Offline¶

Embeddings¶

mlxcel embed -m nomic-ai/modernbert-embed-base \
  -p "search_query: What is vector search?" \
  -p "search_document: Vector search retrieves nearby embeddings." \
  -p "search_document: A compiler translates source code."

Repeat -p for text and --image for supported multimodal inputs. An empty or whitespace-only -p is refused, as mlxcel rerank refuses such a -d. --instruction, --dimensions, --pooling, --embd-normalize, --max-length, --batch-size, --models-dir, and --json expose the same important choices as serving. With several inputs, the command prints cosine similarity for dense vectors or MaxSim for multi-vector outputs.

Reranking¶

mlxcel rerank -m BAAI/bge-reranker-v2-m3 \
  -q "What is vector search?" \
  -d "Vector search retrieves nearby embeddings." \
  -d "A compiler translates source code." \
  --top-n 1

--query-image and repeated --image add visual inputs for a supported multimodal reranker. --instruction, --max-length, --batch-size, --models-dir, and --json are also available.

Serving One or Three Model Kinds¶

# Embedding only
mlxcel-server -m sentence-transformers/all-MiniLM-L6-v2 --port 8080

# Cross-encoder reranking only
mlxcel-server -m BAAI/bge-reranker-v2-m3 --port 8080

# Chat, embedding, and reranking in one process
mlxcel-server \
  -m mlx-community/Qwen3-4B-4bit \
  --embedding-model Qwen/Qwen3-Embedding-0.6B \
  --reranker-model mlx-community/Qwen3-Reranker-0.6B-4bit \
  --port 8080

# A generative reranker alone: both spellings intentionally name the same model
mlxcel-server \
  -m mlx-community/Qwen3-Reranker-0.6B-4bit \
  --reranker-model mlx-community/Qwen3-Reranker-0.6B-4bit

Each embedding and reranking provider owns its model, tokenizer, and MLX arrays on a dedicated thread. It can run beside the chat worker without sharing that worker's stream. One process supports at most one embedding model and one reranker model. --embedding/--embeddings and --rerank/--reranking select restricted llama-server-compatible modes and require the corresponding provider.

Important capacity controls are:

Index Design Is Part of the Model Contract¶

Record at least:

• model repository and immutable revision;
• tokenizer, prefix/instruction, pooling, normalization, maximum length, and output dimension;
• dense or multi-vector shape, dtype, and similarity function;
• chunker version, overlap, text normalization, and parent-document mapping;
• ANN or late-interaction index algorithm and parameters;
• metadata-filter semantics, deletion generation, and tenant boundary.

Very large chunks can dilute the relevant sentence and consume the reranker's token budget. Tiny chunks lose context and multiply index entries. Start from semantic boundaries, add modest overlap, keep stable document and section IDs, and aggregate duplicate parent documents after retrieval.

mlxcel deliberately stops at model inference: it does not store vectors, build an ANN index, implement sparse retrieval, or join metadata. Adapter loading and merging for embedding checkpoints and loading more than one embedding or reranker of the same kind in a process are also outside the current scope.

Evaluation and Capacity Planning¶

Measure retrieval and reranking separately and together. Increase K until recall saturates, rerank each candidate list, and choose the smallest K that reaches the quality target with tail-latency headroom. Choose N from the downstream context or display budget, not from the maximum the reranker accepts.

Offline corpus encoding prefers large batches and retryable jobs; online query encoding prefers small dynamic batches and low tail latency; reranking adds a query-document pair dimension. Do not let a bulk reindex job monopolize the same accelerator used for interactive queries without explicit capacity isolation.

Security and Isolation¶

• Apply tenant and authorization filters before returning candidates; vector similarity is not access control.
• Do not log sensitive text or vectors unless policy permits it. Embeddings can leak attributes and are not anonymous identifiers.
• Limit document length and candidate count so one request cannot monopolize a worker.
• Treat remote and local image references as resource access, not harmless strings; constrain trusted callers, egress, and filesystem visibility.
• Keep untrusted retrieved text separate from system instructions when passing evidence to a generator.
• Version deletions and reindex jobs so removed documents disappear from vector, metadata, and document stores together.

Internal Model Embeddings Are Still Different¶

Generation models also map token IDs to hidden vectors and project image, audio, or video features into a language-model sequence. Those tensors can be token-level, unpooled, position-dependent, and private to a particular generation architecture. The new public embedding subsystem is different: it performs explicit checkpoint detection, attention masking, pooling or multi-vector projection, normalization, batching, length enforcement, usage accounting, and a stable API contract. Do not use a generation model's internal hidden states as a substitute for a retrieval model merely because both are called “embeddings.”

Further Reading¶

• Reimers and Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Khattab and Zaharia, ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
• Hugging Face, Text Embeddings Inference quick tour