Embedding and Reranking APIs¶

mlxcel exposes OpenAI-compatible embeddings, llama-server-native embeddings, and Jina/Cohere/TEI-compatible reranking. These routes use dedicated model workers and are independent of the generation worker.

Route Matrix¶

Rerank response shape is selected by the document-list field, not the route: documents requests a Jina/Cohere envelope, while texts requests a TEI-compatible bare array.

Loading Providers¶

# Embeddings only
mlxcel-server -m sentence-transformers/all-MiniLM-L6-v2 --port 8080

# Cross-encoder reranking only
mlxcel-server -m BAAI/bge-reranker-v2-m3 --port 8080

# Chat plus both secondary providers
mlxcel-server \
  -m mlx-community/Qwen3-4B-4bit \
  --embedding-model Qwen/Qwen3-Embedding-0.6B \
  --reranker-model mlx-community/Qwen3-Reranker-0.6B-4bit \
  --port 8080

-m can auto-detect an embedding checkpoint or a one-label ForSequenceClassification reranker, in which case chat remains unloaded. Generative Qwen3 and Qwen3-VL rerankers cannot be distinguished from generators by config alone and therefore require --reranker-model. For a generative rerank-only server, pass the same checkpoint to both -m and --reranker-model.

POST /v1/embeddings¶

Request¶

{
  "model": "optional served model id",
  "input": ["query: vector search", "passage: nearest-neighbor retrieval"],
  "encoding_format": "float",
  "dimensions": 256,
  "instruction": "optional family-specific query instruction",
  "embd_normalize": 2,
  "user": "accepted and ignored"
}

input accepts five shapes:

The actual JSON spelling for typed parts is:

{
  "input": [
    {"type": "text", "text": "a red chart"},
    {"type": "image_url", "image_url": {"url": "file:///data/chart.png"}}
  ]
}

model, when present, must equal the served embedding model ID. encoding_format is float by default or base64. dimensions must be between 1 and the native output width. instruction is forwarded to family-specific text formatting; families without such a contract ignore it. embd_normalize overrides the server/checkpoint normalization for this request: -1 none, 0 maximum-absolute int16 scaling, 1 L1, 2 L2, and values above 2 select that p-norm.

An image URL may be a data:image/...;base64, URI, file:// URL, HTTP(S) URL, or local path, subject to the shared media limits and process network/filesystem permissions.

Dense Response¶

{
  "object": "list",
  "data": [
    {"object": "embedding", "index": 0, "embedding": [0.01, 0.02, 0.03]}
  ],
  "model": "all-MiniLM-L6-v2",
  "usage": {"prompt_tokens": 7, "total_tokens": 7}
}

Entries preserve request order. Token usage counts non-padding tokens, including special tokens. total_tokens equals prompt_tokens.

Multi-Vector and Base64 Responses¶

A late-interaction model returns embedding: [[...], [...]], one row per real token. With encoding_format: "base64", embedding is standard padded base64 over row-major little-endian f32 bytes. A multi-vector entry also carries "shape": [tokens, dimensions]; a dense vector does not need shape.

POST /embedding and /embeddings¶

The native llama-server routes accept either input or the legacy content field. They run the same model but deliberately return a different schema from /v1/embeddings:

curl -s http://localhost:8080/embedding \
  -H 'Content-Type: application/json' \
  -d '{"content":"hello"}'

[
  {"index": 0, "embedding": [[0.01, 0.02, 0.03]]}
]

The top level is a bare array and embedding is nested one level deeper. Clients expecting the OpenAI object/data/model/usage envelope must call /v1/embeddings.

POST /v1/rerank and Aliases¶

Request¶

{
  "model": "optional served reranker id",
  "query": "What is vector search?",
  "documents": [
    "Vector search retrieves nearby embeddings.",
    "A compiler translates source code."
  ],
  "top_n": 1,
  "return_documents": true,
  "instruction": "optional; generative rerankers only",
  "user": "accepted and ignored"
}

query and each document may be a string or an object with text, image, and/or image_url:

{
  "query": {"text": "find a revenue chart", "image_url": {"url": "query.png"}},
  "documents": [
    {"text": "quarterly revenue", "image": "chart.png"},
    {"image": {"url": "photo.png"}}
  ],
  "top_n": 2,
  "return_documents": true
}

image wins if both image fields are present. An image may be a URL string or {"url": "..."}. An image-only object is valid for a multimodal reranker. Text-only rerankers reject image items. top_n must be at least 1. A sequence classifier rejects instruction because it has no prompt in which to place it.

Jina/Cohere Response¶

Using the documents field selects this envelope:

{
  "model": "bge-reranker-v2-m3",
  "object": "list",
  "results": [
    {
      "index": 0,
      "relevance_score": 0.9948,
      "document": "Vector search retrieves nearby embeddings."
    }
  ],
  "usage": {"prompt_tokens": 31, "total_tokens": 31}
}

document is present only when return_documents or return_text is true. It echoes the original request item. Results are sorted by descending score, ties by ascending original index, and then truncated to top_n.

TEI Response¶

Using texts without documents selects the Text Embeddings Inference-compatible bare array. return_text controls the echo:

{
  "query": "What is vector search?",
  "texts": ["Vector search retrieves nearby embeddings.", "A compiler translates source code."],
  "return_text": true
}

[
  {"index": 0, "score": 0.9948, "text": "Vector search retrieves nearby embeddings."},
  {"index": 1, "score": 0.0012, "text": "A compiler translates source code."}
]

If both documents and texts are sent, documents wins and the Jina/Cohere envelope is returned.

Error Contract¶

Errors use mlxcel's structured error response. Important status codes are:

Server Flags¶

The embedding and reranking workers each own their model, tokenizer, and MLX arrays on a dedicated thread. One process supports one embedding provider and one reranker provider. Queue depth and timeout are shared configuration names but apply to each worker's own queue and request lifecycle.

Client Examples¶

OpenAI Python SDK¶

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="none")
response = client.embeddings.create(
    model="Qwen3-Embedding-0.6B",
    input=["What is vector search?", "Vector search retrieves nearby embeddings."],
)
print(response.data[0].embedding[:8])

Reranking with httpx¶

import httpx

response = httpx.post(
    "http://localhost:8080/v1/rerank",
    json={
        "query": "What is vector search?",
        "documents": [
            "Vector search retrieves nearby embeddings.",
            "A compiler translates source code.",
        ],
        "top_n": 1,
        "return_documents": True,
    },
)
response.raise_for_status()
print(response.json()["results"])

For model contracts, pooling rules, multi-vector MaxSim, evaluation, and index design, read Embeddings and Reranking.