Python Clients¶
Scope¶
Python is a client-side convenience, never an inference-runtime dependency: the engine is the native mlxcel / mlxcel-server binary and every Python path here talks to its HTTP API.
There are two ways to do that. The mlxcel package shipped in python/ wraps the server for you, either spawning and supervising a local one (managed mode) or connecting to one you started (connect mode). Or you skip the package and call the HTTP API directly with the official OpenAI SDK or any HTTP client, which is what the rest of this page shows.
The shipped mlxcel package¶
Python 3.9 or newer, with openai>=1.40 and httpx>=0.27. Managed mode also needs the mlxcel binary, which the client finds through the binary= argument, then MLXCEL_BIN, then mlxcel on PATH.
Managed mode. Pass a model and the client spawns mlxcel serve, waits for the server to report ready, and stops it on exit. The model loads and warms up before the listener binds, so a first run that downloads weights can take a while; the client polls /health with a startup_timeout defaulting to 600 seconds and watches the child for an early exit.
import mlxcel
with mlxcel.LLM("mlx-community/Qwen3-4B-4bit") as llm:
print(llm.generate("def fib(n):", max_tokens=128, temperature=0.7))
print(llm.chat([{"role": "user", "content": "Hello"}], max_tokens=64))
print(llm.model) # resolved model id, auto-discovered from /v1/models
Use the context manager: it guarantees shutdown. A leaked handle is still covered by an atexit cleanup and a best-effort finalizer.
Constructor arguments split in two. binary=, host= / port= (forces TCP), socket= (an explicit Unix socket path), api_key= and startup_timeout= configure the client. Everything else travels through **server_kwargs and configures the spawned mlxcel serve process instead: ctx_size=, n_predict=, alias=, warmup=, and extra_args=[...], which is forwarded verbatim.
with mlxcel.LLM(
"mlx-community/Qwen3-4B-4bit",
ctx_size=8192,
warmup=False,
extra_args=["--parallel", "1"],
) as llm:
print(llm.generate("hello", max_tokens=32))
Connect mode. Pass base_url= or socket= and no model, and nothing is spawned or supervised.
llm = mlxcel.LLM(base_url="http://localhost:8080/v1") # TCP
llm = mlxcel.LLM(socket="/tmp/mlxcel.sock") # Unix domain socket
Passing both a model and a connect target raises MlxcelError, because the mode would be ambiguous. AsyncLLM takes the same arguments and returns awaitables. For anything the wrapper does not expose, llm.openai_client is the underlying openai client, so tools, response_format, logprobs and multimodal all remain reachable.
Sampling parameters. generate and chat pass the OpenAI sampling fields (max_tokens, temperature, top_p, stop, seed, presence_penalty, frequency_penalty, logit_bias, n, logprobs, top_logprobs) as top-level request fields. Server-specific knobs such as top_k, min_p, repetition_penalty, and the DRY settings travel in the request body through the SDK's extra_body, and so does any other unknown keyword (for example p_less=True). An explicit extra_body={...} wins on conflict. top_k at or above the model's vocabulary size keeps every token, the same as 0 (disabled), as in llama-server.
Calling the HTTP API directly¶
The rest of this page uses the OpenAI SDK and httpx against a server you start yourself.
Start the Server¶
mlxcel serve \
--model mlx-community/Qwen3-4B-4bit \
--host 127.0.0.1 \
--port 8080 \
--api-key local-secret
Use mlxcel-server instead when you deploy the standalone server binary. Keep the service on loopback unless remote access has been deliberately secured.
Install Client Libraries¶
openai covers the OpenAI-compatible API. httpx is useful for mlxcel-native helpers such as /tokenize and for endpoints not exposed by the SDK.
Synchronous Chat¶
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:8080/v1",
api_key="local-secret",
)
response = client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "Explain unified memory briefly."}],
temperature=0.0,
max_tokens=128,
)
print(response.choices[0].message.content)
Retrieve the advertised model id with client.models.list() if your deployment does not use default.
Streaming¶
stream = client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "Count from one to five."}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
print()
Closing the response or cancelling the client task disconnects the request; the server observes disconnect cancellation at supported stages.
Async Chat¶
import asyncio
from openai import AsyncOpenAI
async def main() -> None:
client = AsyncOpenAI(
base_url="http://127.0.0.1:8080/v1",
api_key="local-secret",
)
response = await client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "Say hello in Korean."}],
max_tokens=64,
)
print(response.choices[0].message.content)
await client.close()
asyncio.run(main())
Responses API¶
Recent OpenAI SDK releases can call mlxcel's /v1/responses endpoint:
response = client.responses.create(
model="default",
input="Summarize why KV caches matter in one sentence.",
max_output_tokens=80,
)
print(response.output_text)
mlxcel implements the subset documented in Responses API. Do not assume that hosted OpenAI-only tools or every SDK field is available.
Multimodal Chat¶
Use the standard chat content-part shapes documented in the REST API. For example, an inline image can be supplied as a data URI:
response = client.chat.completions.create(
model="default",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image."},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
],
}],
max_tokens=128,
)
The loaded model must implement the requested modality. Image/audio URL and local-path handling follows the server security limitations in the REST manual; an API key does not sandbox network or file access.
Embeddings¶
Start the server with an embedding checkpoint in -m, or add one beside chat with --embedding-model. The OpenAI SDK calls /v1/embeddings directly:
response = client.embeddings.create(
model="Qwen3-Embedding-0.6B",
input=[
"What is vector search?",
"Vector search retrieves nearby embeddings.",
],
)
for item in response.data:
print(item.index, item.embedding[:8])
Use an ordinary HTTP client for mlxcel extensions such as typed image inputs, instruction, or per-request embd_normalize. Multi-vector late-interaction responses contain a matrix rather than one flat vector; base64 responses include a shape field.
Reranking with httpx¶
The OpenAI SDK does not define a reranking resource. Call /v1/rerank with httpx after starting a server with a sequence classifier in -m or a --reranker-model:
import httpx
headers = {"Authorization": "Bearer local-secret"}
with httpx.Client(base_url="http://127.0.0.1:8080", headers=headers) as http:
result = http.post(
"/v1/rerank",
json={
"query": "What is vector search?",
"documents": [
"Vector search retrieves nearby embeddings.",
"A compiler translates source code.",
],
"top_n": 1,
"return_documents": True,
},
)
result.raise_for_status()
print(result.json()["results"])
Using texts instead of documents requests the TEI-compatible bare-array response. See Embedding and Reranking APIs before assuming that the OpenAI, native llama-server, Jina/Cohere, and TEI shapes are interchangeable.
Native Tokenization with httpx¶
import httpx
headers = {"Authorization": "Bearer local-secret"}
with httpx.Client(base_url="http://127.0.0.1:8080", headers=headers) as http:
result = http.post("/tokenize", json={"content": "Hello, mlxcel!"})
result.raise_for_status()
print(result.json())
Use /detokenize for the inverse operation. These native endpoints are not methods of the OpenAI SDK.
Error Handling¶
import openai
try:
client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "Hello"}],
)
except openai.AuthenticationError as exc:
print("check the Bearer API key", exc)
except openai.BadRequestError as exc:
print("request is unsupported or invalid", exc)
except openai.APIStatusError as exc:
print("server returned", exc.status_code, exc)
A full prefill queue is reported as HTTP 503. Configure SDK retries deliberately for your workload; blind retries can amplify overload. See REST API errors for mlxcel error objects.