Continuous Batching¶

Table of Contents¶

• What Is Batching?
• Static vs Continuous Batching
• Iteration-Level Scheduling in mlxcel
• Features
• Per-Sequence KV Cache Isolation
• Chunked Prefill
• Preemptive Eviction
• Tensor-Batched Decode
• Native Batched Decode
• Paged KV Cache and Prefix Sharing
• Decode Headroom Under a Tight Budget
• Token-Exactness Between the CLI and the Server
• Speculative Decoding Under Continuous Batching
• Configuration
• Admission Control
• Monitoring
• Performance Data

What Is Batching?¶

When a language model runs a forward pass, it loads all of its weight tensors from memory into compute units, performs the matrix multiplications, and produces output logits. For a large model, this memory-loading cost is constant regardless of how many tokens are being processed simultaneously.

Batching exploits this property: instead of running the forward pass once per request, multiple requests are grouped together and processed in a single pass. The weight-loading cost is paid once and amortized across all requests in the batch, which increases throughput significantly.

Static vs Continuous Batching¶

Static batching groups requests that arrive at roughly the same time, processes them together from start to finish, and only starts a new batch after the current one completes. This has a critical inefficiency: if one request in a batch finishes early, its compute slot sits idle until every other request in the batch also finishes. Long-running requests hold resources that completed requests no longer need.

Continuous batching eliminates this waste by operating at the level of individual decode steps (iterations) rather than full requests. At each step:

• Requests that completed in the previous step are removed from the batch immediately.
• New requests that are waiting in the queue are added to fill available slots.

The batch composition changes dynamically after every decode step. A server handling 100 requests per minute with continuous batching keeps compute slots occupied continuously, rather than stalling while waiting for the slowest request in each static group.

gantt
    title Static Batching vs Continuous Batching
    dateFormat X
    axisFormat %s

    section Static Batching
    Batch 1 (req A + B + C)  :a1, 0, 8
    Idle (A done early)      :crit, a2, 3, 8
    Batch 2 (req D + E + F)  :a3, 8, 16

    section Continuous Batching
    Request A                :b1, 0, 3
    Request B                :b2, 0, 6
    Request C                :b3, 0, 8
    Request D (joins at t=3) :b4, 3, 9
    Request E (joins at t=6) :b5, 6, 11
    Request F (joins at t=8) :b6, 8, 13

In the continuous batching timeline, as soon as Request A finishes at t=3, Request D fills its slot. There are no idle periods.

Iteration-Level Scheduling in mlxcel¶

The batch scheduler in mlxcel operates on a tick-by-tick basis. On each tick:

1. Drain incoming requests from the channel (non-blocking). New requests are added to the prefill queue.
2. Decide the action for this tick:
3. If a chunked prefill is in progress, interleave a decode step with the next prefill chunk.
4. If the batch has room and the prefill queue is non-empty, admit one new sequence (prefill it).
5. If the batch has active decoding sequences, run a batched decode step.
6. If both the batch and the queue are empty, block until the next request arrives.
7. Execute the chosen action.
8. Finalize completed sequences: release their KV caches and send their accumulated tokens to the waiting HTTP response.

The scheduler decides what runs each tick; the engine runs it. mlxcel_core::engine::Engine owns the model and the KV pool, and every model forward, sampler draw, and finish step goes through Engine::prefill, Engine::step (one entry for every row count, so a lone request is a batch of one), or the pipelined submit / finish_rows pair. A batch of one samples through the per-row sampling chain; a larger batch whose rows share fused-compatible sampling parameters takes one fused [B, vocab] -> [B] draw. The sampled tokens are evaluated through a fallible MLX boundary before they are read, so an MLX error during that evaluation finishes the affected requests with inference backend error instead of aborting the server. Admission, tick policy, preemption, the prompt cache, disaggregated handoff, and the speculative burst generators stay in the scheduler.

The request lifecycle follows this state machine:

stateDiagram-v2
    [*] --> Queued: Request arrives
    Queued --> Prefilling: Admitted to batch
    Prefilling --> Decoding: Prefill complete
    Decoding --> Finished_Stop: EOS token generated
    Decoding --> Finished_Length: max_tokens reached
    Decoding --> Finished_Cancelled: Client disconnected
    Decoding --> Finished_Error: Internal error
    Decoding --> Queued: Preempted (re-queued)
    Finished_Stop --> [*]
    Finished_Length --> [*]
    Finished_Cancelled --> [*]
    Finished_Error --> [*]

Features¶

Per-Sequence KV Cache Isolation¶

Each sequence in the batch has its own KV cache. The cache stores the key and value tensors for every attention head and layer for all previously processed tokens in that sequence. This isolation means that:

• Sequences at different stages of generation do not interfere with each other.
• A sequence can be added or removed from the batch without affecting other sequences' caches.
• The cache for a completed sequence is released immediately, freeing memory for new requests.

Chunked Prefill¶

When a request arrives with a very long prompt, processing the entire prompt in one prefill pass takes proportionally longer. During this time, active decoding sequences receive no decode steps, which increases their time-to-next-token.

Chunked prefill splits the prompt into chunks and processes one chunk per tick, interleaving decode steps for active sequences between chunks. When --prefill-chunk-size is unset, the chunk is 2048 tokens while no other sequence is decoding (preserving single-stream time to first token) and 512 tokens while a decode batch is live (bounding the effect of an admission on running streams). An explicit --prefill-chunk-size, -b/--batch-size (LLAMA_ARG_BATCH), or MLXCEL_PREFILL_CHUNK pins both states. The CLI (generate, run) uses 2048.

flowchart LR
    P1["Prefill\nchunk 1"] --> D1["Decode\nstep"] --> P2["Prefill\nchunk 2"] --> D2["Decode\nstep"] --> dots["..."] --> FT["First\ndecode token"]

This ensures that long prompts do not starve active sequences. The cost is that the time-to-first-token for the new request increases proportionally to the number of chunks, but active requests stay responsive.

Measured with four Llama-3.2-1B 4-bit streams decoding while an 8192-token request is admitted, a 2048-token chunk gave about 128 ms stream ITL p95 and about 2 s admitted-request TTFT, while 512 gave about 56 ms ITL p95 and about 4 s TTFT; the adaptive default takes 512 only while that tradeoff applies.

Every prefill path (CLI and server) builds one prefill plan: an adopted prompt-cache prefix is skipped, a snapshot family's chat prompt is split once at its history boundary, the rest is cut into chunk pieces, and a piece is tile-padded on M5 and later hardware only where the model opts in.

Set --prefill-chunk-size 0 to disable chunking entirely. In that mode, prompts are prefilled in a single pass, which may be preferable for workloads with many short prompts where the chunking overhead is not worth the latency benefit for existing sequences.

Preemptive Eviction¶

When --enable-preemption is set and the batch is full, a high-priority incoming request can evict an active lower-priority sequence to gain a slot:

1. The scheduler selects a victim sequence from the active batch according to the preemption policy.
2. The victim's generated tokens are saved, and its KV cache is released.
3. The victim is re-queued in the prefill queue. It will be re-admitted and re-prefilled when a slot becomes available.
4. The high-priority request is prefilled immediately.

The preemption policy controls which active sequence is selected as the victim:

Preemption is disabled by default because evicting and re-prefilling a sequence wastes compute. Enable it only when low-latency handling of high-priority requests is more important than overall throughput.

Tensor-Batched Decode¶

When multiple sequences are in the active batch, mlxcel runs a single forward_batched() call rather than separate forward() calls per sequence. MLX uses lazy evaluation: the computation graph is built for all sequences together and evaluated in a single eval() call.

This matters because model weight tensors need only be loaded from memory once per batch, regardless of batch size. The weight-loading cost, which is the dominant cost for memory-bandwidth-bound inference, is amortized across all active sequences.

For models with explicit forward_batched() implementations (such as Llama 3), the batching is even more efficient:

1. Embedding lookup, normalization, Q/K/V projection, output projection, and FFN are run in a single batched operation.
2. Attention is computed per-sequence (required because each sequence has a different KV cache offset and RoPE position state).
3. The final norm and language model head are batched again.

This split-attention pattern maximizes the amortization benefit while respecting the per-sequence cache structure.

Batched decode is pipelined: the scheduler submits the next step before it reads back the current tokens, so CPU-side bookkeeping overlaps GPU work. Requests that carry logit_bias or ignore_eos stay on this lookahead pipeline; their per-row token bias is added to the logits before the fused sampling dispatch, and rows without a bias are left bitwise unchanged.

Recurrent and linear-attention families (Mamba, Mamba 2, RWKV 7, Jamba, Falcon-H1, Granite MoE Hybrid, Kimi Linear, RecurrentGemma and others) keep their state in model-owned storage keyed by sequence id, and batch like any other family. A model whose supports_batching() returns false (for example DiffusionGemma, LLaDA2-MoE, Florence-2, Nemotron-Parse, Nemotron VoiceChat, Falcon-OCR and Phi-4 multimodal) is clamped to a batch of one, and the server logs does not support batched decode; clamping max_batch_size. Block-diffusion models (DiffusionGemma) are served by a dedicated serial worker and never join the batch scheduler; see Block Diffusion.

Native Batched Decode¶

A model that batches runs one of two multi-row decode paths. BatchedDecodeKind::Native shares the model weights in one forward graph for all rows, so aggregate decode throughput grows with the batch. BatchedDecodeKind::PerRowFallback is the trait default: the rows are batched in the scheduler but each runs its own forward pass, so aggregate throughput does not scale with --parallel. At worker startup the server logs which path the loaded model uses:

Model <id> decode path: native batched forward
Model <id> decode path: per-row fallback (aggregate decode throughput will not scale with --parallel)

Native batched decode covers dense, MoE and MLA transformers, Gemma 3n and Gemma 4, GPT-OSS and other families with model-owned caches, recurrent and linear-attention families, Qwen VL and other VLM wrappers (which delegate to their text tower), and most tensor-parallel runtimes (Llama, Qwen 3, Qwen 3.5, Gemma 3, ERNIE 4.5, Hunyuan V1 Dense). A few families still take the per-row path: Baichuan, Granite 4 Vision, Llama 4 (C++ model), Muse Glimmer text, Qwen4-Exp, the pipeline-parallel server model and the tensor-parallel Gemma 4 model. The pinned per-family classification is src/models/batched_decode_coverage_tests.rs.

Batched decode has measured gaps. On M1 Ultra, B=1 decode is 3 to 6% lower than before the batched decode work on a few checkpoints, and Mixtral stays below its B=1 rate at B=8 with reusable prefixes in the prompt cache. Both are tracked as Known Issue lablup/mlxcel#2817. On M1 and M2, dense products of 2 to 6 rows stay on GEMV; MLXCEL_DENSE_GEMV_ROWS=0 restores the upstream dense dispatch.

Paged KV Cache and Prefix Sharing¶

Under --decode-storage-backend paged, the server keeps both the cross-request prompt-prefix cache and per-sequence decode state in one refcounted, copy-on-write block pool. Two requests that share a prompt prefix store that prefix's KV blocks once: the second request adopts the first request's blocks by reference instead of re-prefilling them, and a block forks (copy-on-write) only when one sequence's content diverges from a shared block.

Paged adopt and donate apply to the pool-backed FP16 families (dense-natural backends such as Qwen 3 and Llama 3). Model-owned-state families (Gemma 3, Llama 4, Qwen 3.5) and recurrent or hybrid SSM models keep dense or model-owned caches and stay out of the pool.

The block pool is bounded by --kv-cache-budget <BYTES|auto|none> (env MLXCEL_KV_CACHE_BUDGET) and defaults to auto, which derives a cap from available memory, weights, and activation headroom. A byte count sets an explicit cap; none or 0 opts out and leaves the pool unbounded. Under a budget the scheduler evicts cold cached prefixes first, then preempts, and only then rejects a request. GET /v1/cache/stats reports paged block usage: block size, allocated, live, free, bytes reserved and in use, and the budget.

A request that adopted a cached prefix is charged only for the blocks its suffix will add, since the adopted blocks are already in use. Decode reserves its blocks before the forward. When a running row needs a block and none is free, the reclaim order is: evict a cold prompt-cache prefix, then drop a queued request's adopted prefix (it re-prefills cold when admitted), then preempt the least-progressed running row.

Measured on M1 Ultra with qwen3-0.6b-4bit (about 112 KiB of KV per token): a 1024-token prefix shared by 8 requests stores once instead of eight times, saving roughly 784 MiB, and each adopting request skips the shared prefill entirely while decoding byte-identically to a cold run. Batched paged decode uses a native block-table kernel that runs 1.9x faster than the gather reference at a 512-token prompt and 10.9x faster at 4096 tokens (batch 4); the gap widens with context because the gather path re-materializes the visible window every step.

Decode Headroom Under a Tight Budget¶

When the budget is small relative to the concurrent work, admitting a request into exactly the free blocks leaves the running rows no room to grow, so the next decode ticks preempt the least-progressed row and it re-prefills from scratch when admitted again. --kv-admission-watermark F (env MLXCEL_KV_ADMISSION_WATERMARK, range 0.0 to 0.5, default 0.01) keeps the fraction F of the block budget free at admission: a request is admitted only when the free blocks minus F * budget cover it. To make that room, admission evicts cold prompt-cache prefixes and, for a higher-priority request, preempts strictly lower-priority rows before it waits. The watermark applies only while a decode batch is live, so a request that fits the budget is never refused; it waits for running rows to finish instead. 0 disables it, and it has no effect without a block budget.

Measured on GB10 with meta-llama-3.1-8b-instruct-4bit, --kv-cache-budget 600000000 (4444 blocks), --max-batch-size 4, and three concurrent 3-turn conversations of 600-token generations:

The default is the smallest value on that grid that lowered preemptions without losing a turn. Raise it when a tight budget preempts often and the extra queueing latency for new requests is acceptable.

Token-Exactness Between the CLI and the Server¶

mlxcel run (with and without -p) and its chat REPL start the server's model worker in-process at one slot and submit each turn through the /v1/chat/completions request path (see CLI Reference). For the same chat request under MLXCEL_SDPA_DETERMINISTIC=1, greedy output from mlxcel run -p and from the server is identical. make engine-parity MODEL=<dir> checks this invariant; see Testing.

Two server features change near-tie greedy tokens and are measured divergences rather than part of the invariant:

• A prompt-cache hit forwards only the suffix after the cached prefix, which is a different prefill partition. It reproduces the cold run exactly only when the adopted prefix ends on a split point of the cold plan (the history boundary of a snapshot family, or a chunk edge). The chat REPL runs with the cache on; run -p runs with it off.
• Continuous batching with other live sequences.

To reproduce the single-request path on the server, use --max-batch-size 1 or --decode-storage-backend dense, plus --no-prompt-cache. A client can also send cache_prompt: false per request; see Reproducible Greedy Output.

Speculative Decoding Under Continuous Batching¶

DFlash, MTP, and prompt-lookup speculation run inside the batch scheduler. For prompt lookup (--draft-kind prompt-lookup), each proposing row runs one verify round alone on a tick while the other rows take the batched step, and verify rounds run only while the active batch has at most --prompt-lookup-max-batch rows (default 2), with no unified KV budget (--kv-unified disables them), and on paged storage only while the total context is at most 4608 tokens. Gemma 4 batched MTP (MLXCEL_ENABLE_MTP_BATCH) is byte-identical to per-sequence decoding on row-wise geometries; it runs on CUDA for windows of up to 4 rows within the sliding window, and Metal and ROCm decline it. See Speculative Decoding and MTP and DFlash.

Configuration¶

The following mlxcel serve flags control the batch scheduler:

With continuous batching enabled, an explicit --ctx-size C is a total server context budget shared by active slots, matching llama.cpp server behavior. The per-slot window is floor(C / --max-batch-size) when --max-batch-size is set, otherwise floor(C / --n-parallel). --no-batch uses one slot and keeps the full explicit context budget. /slots and /health report this effective per-slot context size, and startup rejects values below 512 tokens per slot.

Example: High-concurrency configuration:

mlxcel serve \
  --model ./models/Qwen2.5-7B-Instruct \
  --max-batch-size 8 \
  --max-queue-depth 64 \
  --prefill-chunk-size 512 \
  --enable-preemption \
  --preemption-policy lowest-priority

This example pins the prefill chunk at 512 in both scheduler states; omit --prefill-chunk-size to keep the adaptive default.

Example: Low-latency single-request configuration:

mlxcel serve \
  --model ./models/Llama-3.1-8B-Instruct \
  --max-batch-size 1 \
  --prefill-chunk-size 0

Or equivalently:

mlxcel serve \
  --model ./models/Llama-3.1-8B-Instruct \
  --no-batch

Admission Control¶

When the prefill queue reaches --max-queue-depth, the server stops accepting new requests until queue space is available. Requests that arrive when the queue is full receive HTTP 503 Service Unavailable:

{"error": {"message": "Server is busy. Retry later.", "type": "overloaded"}}

This back-pressure mechanism prevents unbounded memory growth from queued requests. Set --max-queue-depth based on the expected traffic pattern and acceptable queuing latency for your deployment.

On Metal, text-only Nemotron-H also checks each request against a memory estimate and answers HTTP 503 request_memory_budget_exceeded when it does not fit; see Nemotron-H Request Memory Admission.

Monitoring¶

The health endpoint includes batch state when the scheduler is active:

curl -s http://localhost:8080/health | jq .

{
  "status": "ok",
  "batch": {
    "active_sequences": 3,
    "queued_requests": 1,
    "total_completed": 412
  }
}

The /slots endpoint (enabled by default) shows per-slot status:

curl -s http://localhost:8080/slots | jq .

The /metrics endpoint (enabled with --metrics) exposes Prometheus-compatible counters:

curl -s http://localhost:8080/metrics | grep mlxcel_batch

Performance Data¶

The following throughput measurements were taken on an Apple M1 Ultra (128 GB) to illustrate the benefit of batching at different batch sizes. C=4 means 4 concurrent requests; C=8 means 8 concurrent requests.

Scaling observations:

• Small models (under 1B parameters) benefit the most from batching because weight-loading amortization is the dominant factor.
• Medium models (1-4B parameters) show good scaling up to batch sizes of 4-8.
• Large models (7-8B parameters) see moderate improvement because they are already memory-bandwidth-bound at batch size 1.
• Time-to-first-token (TTFT) increases roughly 3-6x when going from 1 to 8 concurrent requests, because prompts are prefilled sequentially. Chunked prefill reduces the impact on active decode sequences but does not eliminate the TTFT increase for newly arriving requests.