OpenAI Responses API¶

mlxcel exposes the OpenAI-compatible Responses API on both mlxcel serve and mlxcel-server. The endpoint is useful when a client already uses the OpenAI Python or TypeScript SDK's client.responses.create(...) surface and you want to point it at a local mlxcel model server.

Endpoints¶

Aliases without /v1 are also mounted: /responses, /responses/{id}, /responses/{id}/cancel, /responses/compact, and /responses/input_tokens.

Quick Example¶

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="sk-no-auth")
resp = client.responses.create(
    model="Qwen3.5-0.8B-4bit",
    input="Reply with: hello",
    max_output_tokens=64,
)
print(resp.output_text)

curl http://127.0.0.1:8080/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.5-0.8B-4bit","input":"Reply with: hello","max_output_tokens":64}'

Supported Request Surface¶

Unsupported built-in tool types such as web_search, file_search, computer_use_preview, code_interpreter, image_generation, mcp, custom, apply_patch, and function_shell return 400 invalid_request_error because mlxcel does not run external hosted tools.

Background Mode¶

POST /v1/responses with "background": true returns HTTP 200 at once with a queued response and runs the generation as a detached job. Poll GET /v1/responses/{id} until the status is terminal.

curl -s http://127.0.0.1:8080/v1/responses -H 'Content-Type: application/json' \
  -d '{"model":"local","input":"Write a haiku about rain.","background":true}'
# {"id":"resp_...","status":"queued","background":true,"output":[],...}
curl -s http://127.0.0.1:8080/v1/responses/resp_...

Every object of a background response carries "background": true; a foreground response has no such key. Until the job is terminal, output is [] and usage is all zeros, so polling never exposes partial output.

What create answers:

• 400 invalid_request_error for everything a foreground request is refused for before generation (malformed fields, unsupported media, invalid sampling or tool parameters, an invalid schema, an unknown previous_response_id), and also when stream is true, when the response store is disabled, or when --responses-background-max-jobs, the response store entry limit, or --responses-store-max-bytes is 0. No job is created.
• 429 with error.code: "rate_limit_exceeded" when --responses-background-max-jobs jobs are already active, or when the active jobs fill --responses-store-max-bytes. Retry after a job finishes.
• 503 when the generation queue is full, as for a foreground request.

Dropping the create connection or a poll cancels nothing; only the cancel endpoint stops a job. A store: false job is allowed, and its final object stays readable by polling for --responses-background-ephemeral-ttl-secs while remaining unusable as previous_response_id. Chaining with previous_response_id to a job that has not finished returns 400 with previous response '<id>' has not finished (status: <status>); once the job is terminal it chains like any stored response. conversation history is read when the job is created and the turn is appended when the job finishes with output, so a failed job appends nothing. context_management runs inside the job, so a compaction failure ends the job as failed instead of failing the create call.

The structured-output constraint is built at create, so an invalid schema is a 400 rather than a failed job. Jobs live in memory only: stopping or restarting the process loses queued and running jobs, and their ids return 404 afterwards. If the generation worker exits while the process keeps running, its jobs end as failed.

Compaction¶

POST /v1/responses/compact takes the full history in input and returns an object of type response.compaction with id, created_at, model, output, and usage; the usage counts the summary generation. output lists the kept system and developer messages, the retained items, and one compaction item. Nothing is stored.

• The retained part is the latest user message plus the newest whole turns that fit keep_tokens (default 1/16 of the context window, at most 8192). A split falls only before a user turn or after a completed tool group, so a function call is never separated from its output.
• The summary is one non-streaming generation on the /v1/responses worker path. max_output_tokens bounds it (default 1024, from 1 to 16384).
• stream, previous_response_id, and conversation are rejected, and keep_tokens must be above 0.
• A history longer than one context window, a summary that does not fit, or a summary that does not shorten the context returns 400 with code context_length_exceeded and changes nothing.
• The compaction item's encrypted_content is base64url JSON holding the summary and the retained items. It is not encrypted and is treated as client data, like any input array.

/v1/responses and /v1/responses/input_tokens accept the compaction input item and replay it: the summary stands in for the earlier turns, and previous_response_id chains replay it too.

With context_management: [{"type": "compaction", "compact_threshold": N}] (N from 1 to 100000000), /v1/responses counts the rendered input the way /v1/responses/input_tokens does. When the count reaches N or leaves no room for output, the history is compacted as above before generation, and output starts with the compaction item. A stored response replays the summary on the next previous_response_id hop without compacting again below the threshold. A streamed request sends two extension events after response.in_progress, before the compaction item at output_index 0:

For a request that carries context_management, generation is capped at the context left after the prompt, and a length stop returns status: "incomplete" with reason max_output_tokens. Compaction finishes before the stream opens, so a compaction failure is an HTTP error.

Storage Flags¶

Both server entry points accept the same store controls:

Active background jobs sit outside the TTL and LRU limits, so a running job is never evicted. Their approximate bytes count against --responses-store-max-bytes when the job is admitted, and a job that would push the active total past the budget is refused; they never evict stored responses. A finished job's final object becomes a stored response. If that object alone is larger than the budget, a failed response with error.code: "server_error" and empty output is kept so a poll still gets an answer. The final object of a store: false job is kept for --responses-background-ephemeral-ttl-secs only and is never usable as previous_response_id. Background jobs live in memory and a restart loses them. Background mode is disabled when the job limit, the entry limit, or the byte budget is 0.

When response persistence is disabled, GET, DELETE, cancel, and previous_response_id requests return 400. When the conversation store is disabled, requests using conversation return 400.

Streaming Events¶

Streaming responses are sent as typed SSE frames. The common event sequence is:

1. response.created
2. response.in_progress
3. response.output_item.added
4. response.content_part.added
5. response.output_text.delta (zero or more)
6. response.output_text.done
7. response.content_part.done
8. response.output_item.done
9. response.completed

Function-call arguments and reasoning traces use the matching response.function_call_arguments.* and response.reasoning_text.* events. Arguments stream as response.function_call_arguments.delta fragments while they are generated for Hermes/Qwen JSON, Mistral ([ARGS] and the JSON list form), Llama 3 JSON, and Qwen3-Coder XML; other tool-call formats send the whole call at the end of generation. The end-of-generation parse stays authoritative, so a streamed call keeps its id and receives the unsent rest of its arguments. On templates whose generation prompt already opens a reasoning block (ending in <think>\n), the reasoning streams as response.reasoning_text.delta in a reasoning item ahead of the message from the first token.

While the prompt cache is on, the server remembers the reasoning item it returned with each plain reply. When a later request replays that assistant turn without its reasoning item, the server re-injects the reasoning so the turn renders the way it was generated and the prompt cache still matches. MLXCEL_REASONING_ECHO_MAX_BYTES bounds the store (default 16 MiB; 0 disables it).