Prompt Cache: Design and Operator Guide¶

This document describes mlxcel's prompt-prefix KV cache. It covers motivation, architecture, cache key semantics, configuration, observability, troubleshooting, and current limitations.

Motivation¶

Large language model inference has two steps: prefill (computing KV activations for every prompt token) and decode (sampling one token per step). Prefill is compute-bound and grows linearly with prompt length. For workloads where many requests share a common prefix (system prompts, tool schemas, conversation history), the server was re-computing the same KV tensors on every request.

Prompt-prefix caching solves this by storing the completed KV cache of a finished sequence and making it available to future requests that start with the same prefix. When a match is found the new sequence adopts the cached tensors and begins prefill only from the first non-matching token, saving the compute that would otherwise reprocess the common prefix.

Architecture Overview¶

flowchart TD
    HTTP["HTTP request"] --> RL["Route layer<br/>(chat_req)"]
    RL -->|"PromptCacheRequestContext"| ENQ["Enqueue to prefill queue"]
    RL -->|"template_sig, session_key, mm_digest"| STORE["PromptCacheStore (Arc&lt;RwLock&lt;Inner&gt;&gt;)<br/>HashMap&lt;digest, entry&gt; + per-bucket RadixTrie"]
    ENQ -->|"compose key"| ADOPT

    subgraph BS["BatchScheduler"]
        direction TB
        ADOPT["try_adopt_cached_prefix()<br/>1. compose PromptCacheKey from context + tokens<br/>2. lookup_longest_prefix() in store<br/>3. take_detached() → DetachedCacheSet<br/>4. CachePool::adopt() → new SequenceId<br/>(KV tensors installed; prefill from matched_len onward)"]
        RUN["prefill suffix tokens → decode loop"]
        DONATE["donate_finished_sequence_cache()<br/>1. CachePool::detach() → DetachedCacheSet<br/>2. build CacheEntry(tokens, detached, bytes)<br/>3. store.insert(key, entry)"]
        ADOPT --> RUN --> DONATE
    end

    STORE -->|"lookup_longest_prefix()"| ADOPT
    DONATE -->|"donate back"| STORE

Key components¶

Cache Key Composition¶

A cache entry is identified by a 32-byte BLAKE3 digest of seven dimensions. A mismatch in any dimension produces a different digest and results in a cold prefill.

Dimensions¶

digest = BLAKE3(
    "mlxcel:prompt-cache:v3"   ← domain separator
    model_id                    ← length-prefixed UTF-8
    lora_id                     ← length-prefixed UTF-8 (empty if None)
    template_sig                ← length-prefixed UTF-8
    session_key                 ← length-prefixed UTF-8
    rope_regime                 ← presence byte + tag byte
    mm_digest                   ← 32 raw bytes (MultimodalDigest)
    prefix_len                  ← u64 little-endian
    tokens[..prefix_len]        ← each token as i32 little-endian
)

model_id¶

The display model identifier returned by AppState::display_model_id. For HuggingFace-style model directories this is the last path component. It ensures that caches from different loaded models never cross-contaminate.

lora_id¶

Empty for the base model. When LoRA adapters are served unfused, it is a digest of the effective adapter scales, so a POST /lora-adapters change never reuses KV computed under other scales. A cache entry built with no adapter cannot be adopted by a request using a LoRA adapter.

template_sig¶

A 64-char lowercase hex BLAKE3 digest of the rendering pipeline inputs:

• chat_template_source: the raw Jinja template string. Any whitespace change invalidates the cache.
• chat_template_kwargs: merged per-request and server-default kwargs, canonicalized (keys sorted) so map reordering does not cause spurious misses.
• tool_choice: effective mode string ("auto", "none", "required", or the function name for "specific").
• tools_digest: order-preserving BLAKE3 digest of the tools array including each tool's type, function name, description, and parameters.

Implication: adding a tool schema to a system prompt that previously had none changes template_sig and forces a cache miss. This is correct because the token stream changes when tools are present.

session_key¶

Resolved from the request's prompt_cache_key, user, or the anonymous sentinel __mlxcel_anon__. Precedence:

1. prompt_cache_key in extra_body (first non-empty wins)
2. user field in the request
3. __mlxcel_anon__ (shared fallback bucket)

The lookup algorithm is two-tier: it first looks for the best match within the exact session_key, then falls back to cross-session candidates from the same (model, lora, template) bucket. A cross-session result is returned only when it is strictly longer than the best in-session result.

rope_regime¶

A one-byte tag for models that pick a RoPE frequency table from the prompt length (Phi-3 and Phi-4 LongRoPE use short_factor up to original_max_position_embeddings and long_factor above it). Two requests share KV only when they get the same tag, taken from the full prompt length of the request. Models whose rotation does not depend on sequence length have no tag.

mm_digest¶

A 32-byte BLAKE3 digest of all resolved image and audio bytes in the request, computed in message-arrival order. For text-only requests this is BLAKE3("mlxcel:prompt-cache:multimodal:v1" || 0u64 || 0u64), a stable constant distinct from any real payload.

Two requests that embed different URLs pointing to the same image bytes produce the same mm_digest (digests are computed over post-decoded bytes). Swapping the order of two images produces a different digest.

Token prefix¶

The first prefix_len token IDs from the fully tokenized prompt (after chat template rendering). The digest covers exactly these tokens; the store then performs longest-prefix matching so a stored entry of length N can serve a request of length >= N that begins with the same N tokens.

Prefix-Stability Guarantees¶

For cache reuse to be safe the token stream must be deterministic across requests that share the same prefix.

preserve_thinking = true default¶

When the prompt cache is enabled, preserve_thinking defaults to true for models that support multi-turn thinking retention (Qwen3.6 and derivatives). Without this default, the chat template renders differently across turns. The second turn's token stream would diverge from the first, making the stored prefix unusable. Operators can override this per-request via chat_template_kwargs.

template_sig invalidation¶

Any change to the chat template string, its keyword arguments, or the tool schema is reflected in a new template_sig digest. Old entries remain in the store but will never match the new digest and will age out via LRU or TTL eviction.

Tool-call caveats¶

Tool responses embed non-deterministic content (e.g., API call results and generated assistant text) which changes the token stream. Only the prefix before any tool call result is likely to match. Enabling the cache with heavily tool-using workloads may see lower hit rates than a system-prompt-only scenario.

Configuration¶

All knobs are available as CLI flags and as environment variables. When both are provided the CLI flag takes precedence.

Flags and environment variables¶

MLXCEL_PROMPT_CACHE_ENABLED is checked before the LLAMA_ARG_CACHE_REUSE alias when the CLI flag is absent. This allows deployments that already use llama.cpp-compatible env vars to control the feature without code changes.

Model-aware capacity defaults¶

When neither the environment variable nor the CLI flag sets a store's byte budget, startup sizes it from the loaded model's config.json. An explicit value is never replaced, and the result is a cap on what the store may hold, not an allocation: resident memory grows only as entries are inserted.

The snapshot store (--prompt-cache-snapshot-capacity-bytes) is raised for every snapshot-only family: the recurrent ones (Qwen 3.5/3.8, Jamba, Falcon-H1, LFM2, Bailing linear, Nemotron-H, Mamba, Granite 4 hybrid, PLaMo 2) and the attention-cache ones (Gemma 3, Gemma 4, Gemma 4 unified, Llama 4, AFMoE, Muse Glimmer, Inkling). The default is max(536870912, min(6 * entry, left / 4)), raised to one entry when that entry fits in left / 2. entry is the snapshot size at a representative length of min(--ctx-size, 8192) tokens (8192 when --ctx-size is unset), and left is the memory that remains after the model's own estimated footprint from the memory estimator. Sliding-window families (Gemma 3, Gemma 4) are sized with every layer at full length, because the history-boundary snapshot is taken before the sliding layers wrap. Both the top-level and the text_config model_type are matched with a trailing _text ignored, so VLM-packaged checkpoints get the same default as their text models.

The KV store (--prompt-cache-capacity-bytes) gets the same treatment for dense-KV families: six window-capped entries at the same representative length, the same ceiling and one-entry floor, and never below the 2 GiB fallback. On Llama 3.1 8B the default becomes 6 GiB. Snapshot-only families skip this store because they never donate into it.

Startup logs Applied model-aware prompt-cache snapshot capacity default or Applied model-aware prompt-cache KV store capacity default at INFO with the previous and recommended capacity, the representative entry size, and the ceiling. When a single entry is larger than its whole store it can never be kept, every turn reports cached=0, and the server logs one WARN per store with the entry size and the capacity, naming --prompt-cache-snapshot-capacity-bytes or --prompt-cache-capacity-bytes.

Quickstart¶

# Enabled by default: model-aware budget (at least 2 GiB), 1024 entries, 1-hour TTL
mlxcel-server -m /path/to/model

# APC is also enabled by default; disable it independently when comparing behavior
mlxcel-server -m /path/to/model --apc-enabled=false

# Larger deployment: 8 GiB, 4096 entries, 30-minute TTL, APC with blake3
mlxcel-server -m /path/to/model \
    --prompt-cache-enabled \
    --prompt-cache-capacity-bytes 8589934592 \
    --prompt-cache-max-entries 4096 \
    --prompt-cache-ttl 1800 \
    --apc-enabled \
    --apc-hash blake3

Tuning guidance¶

When to raise capacity-bytes: when the Prometheus gauge mlxcel_prompt_cache_bytes approaches the configured limit and mlxcel_prompt_cache_evictions_total{reason="lru"} is growing fast. Raising the capacity reduces eviction churn.

When to raise max-entries: when you have many short conversations (each has a small cache entry in bytes but the entry count is the binding constraint). Check mlxcel_prompt_cache_entries against the configured maximum.

When to lower ttl: for workloads where conversations are short-lived and memory is scarce. Lowering TTL removes stale entries proactively before they are pushed out by LRU pressure.

When to raise min-prefix: when your prompts are highly dynamic with short shared prefixes. Raising the threshold prevents the cache from filling up with tiny entries that provide little compute savings relative to the adopt/donate overhead.

Session Snapshots¶

Some families cannot restore from the block-granularity prefix cache at all, because their state is not a plain KV table: Gemma 4, Qwen 3.5, and the SSM hybrids carry recurrent state that only a whole-cache snapshot can reproduce. For them, multi-turn reuse runs through a session snapshot store, keyed by session, and it is what makes a second turn skip the history prefill.

Prefill donates a snapshot at the history boundary. When a request's prompt splits into a rendered conversation history plus a new user message, prefill takes a snapshot at that boundary rather than only at the end of the prompt. The next turn's history is a prefix of the stored one, so it restores instead of recomputing. Without this donation a snapshot-only family never hits on turn two, which is why the behavior is on by default; MLXCEL_DISABLE_BOUNDARY_SNAPSHOT=1 turns it off for attribution runs.

The next turn's prefix is warmed in the background. After a reply finishes, a background task extends the session's boundary snapshot to cover the assistant reply that was just produced, so the next turn prefills only the new user message instead of the reply plus the message. This is separate from the boundary donation because the boundary snapshot is foreground work on the request path and the warm-up is not; MLXCEL_DISABLE_CACHE_WARMUP=1 turns off the warm-up alone.

A session's snapshots form a chain, and a newer one supersedes its ancestors. Each turn extends the chain rather than adding an unrelated entry, so the store holds one live snapshot per session rather than one per turn, and the budget above is a budget over sessions. The scheduler exposes the remaining budget so an operator can see how close the store is to eviction.

A divergent restore is rejected, and the rejection is classified. When a request's history does not match the stored snapshot, restoring from it would produce state that never existed, so it is refused and the request prefills normally. The prompt-cache stats break these rejections out by cause rather than folding them into the miss count, so a session whose client rewrites history looks different from a session that simply arrived cold. Gemma 4 is the exception that restores partially: it can resume at the longest common prefix between the request and the snapshot instead of discarding the whole entry.

Set MLXCEL_APC_TRACE=1 to get one log line per decision, naming which snapshot a request matched and why a divergent one was rejected.

On Metal, text-only Nemotron-H requests that run under request memory admission (see Server) prefill cold: they neither restore nor donate snapshots and skip the background warm-up, while retained snapshots stay counted in the resident memory baseline.

qwen4_exp (Qwen3.8-Flash-Next) snapshots the whole per-sequence state: the GatedDeltaNet recurrence, the PLE window and n-gram contexts, the KV cache and the QSA index, at their stored dtypes. The n-gram row table is never part of a snapshot, so a resident PLE store and an external one restore alike. Restore fails with a named error on a foreign family, layer count or kind, a shape mismatch, or a different KV mode. A request with images or video declines the snapshot with a named reason in the log and prefills cold, so MRoPE state is never stored. A capture whose attention offset differs from the claimed token length is refused, and a restore installs an empty rollback log, so a rewind past the restore point is refused rather than guessing. The snapshot store re-checks the macOS memory reserve before it grows, as described in macOS Memory Guardrails.

Reproducibility and Cache Hits¶

A prompt-cache hit is a split prefill: the cached prefix was computed in an earlier forward and only the remainder is prefilled now. The split changes the floating-point summation order (quantized matmul kernels choose their path by row count, and attention sees a different query shape), so the logits of the first generated token are close to a cold prefill's but not always bit-identical, and a greedy reply can differ from the cold one at a near-tie token.

A hit reproduces the cold run of the same prompt exactly when its adopted prefix ends on one of the cold prefill plan's split points (the history boundary of a snapshot family, or a chunk edge) and the adopted rows were written by the same pieces. Cache off and a cold prefill with the cache on are identical for every family without a history-boundary split.

When a request needs output that does not depend on cache state (regression tests, evaluation runs, comparing two builds), send cache_prompt: false. The field is read at the request root, through the flattened OpenAI-SDK extra_body, or nested inside extra_body, on the chat routes and on native /completion. The request is prefilled cold and neither reads nor donates cache entries, so it leaves the store unchanged for other requests. --no-cache-prompt (or --prompt-cache-enabled=false) does the same server-wide.

Automatic Prefix Caching (APC)¶

APC is a layer on top of the whole-prefix cache, enabled by default (--apc-enabled). Instead of matching only entries whose complete token prefix matches from the start, APC splits each token sequence into fixed-size blocks and builds a chained hash for each block. A future request that shares the first K blocks with a stored entry can reuse those K blocks' KV state even if the total prefix lengths differ.

How it works¶

Each entry stores a BlockHashChain: a sequence of 32-byte hashes where block n's hash is derived from (parent_hash_of_block_n-1, tokens_in_block_n, multimodal_digest). This Merkle-DAG construction means a change to any token propagates to every downstream block hash, so the server can determine exactly how many leading blocks of a new request are consistent with a stored candidate.

At lookup time, after the trie finds a candidate entry, the server recomputes the request's block hashes for the covered prefix and walks both chains in lockstep. The last block index where both chains agree becomes the new matched prefix length (clamped to a full block boundary). This gives finer-grained reuse than whole-prefix matching for workloads with long shared but not-identical prefixes, such as requests that share a system prompt and first few user turns but diverge at different points.

Hybrid SSM models are automatically excluded¶

Models with recurrent layers (Mamba, Mamba2, Jamba, Nemotron-H, GatedDeltaNet/Qwen3-Next, KimiLinear) carry a hidden state in addition to the KV cache. Restoring only the KV tensors while leaving the SSM state at zero produces incorrect outputs. The server detects these model families from config.json at startup and silently disables APC for them. The whole-prefix cache is unaffected and continues to work.

Affected model_type strings: jamba, mamba, mamba2, nemotron_h, gated_delta, kimi_linear, kimi_k3, qwen3_next, falcon_mamba, longcat_flash, longcat_flash_ngram, rwkv7, recurrent_gemma, baichuan_m1. A text_config.model_type and architecture names are matched too.

APC observability¶

GET /v1/cache/stats includes these APC-specific fields:

POST /v1/cache/reset clears the store (including all APC chains) and reports the freed entries and bytes.

When to enable APC¶

APC pays off when your workload has many requests that share a prefix of varying length, long enough to cross several block boundaries but not identical from end to end. A typical example is a shared system prompt followed by different conversation lengths: without APC, each conversation caches its own full sequence and reuse requires an exact prefix match; with APC, any request that shares the leading N blocks can inherit those N blocks' KV state.

APC adds a small per-block hashing cost at insert and lookup time. For text-only workloads on Apple Silicon this is negligible. If you observe latency increases on short requests, check apc_active_entries and consider raising --apc-block-size to reduce the number of blocks computed per request.

Observability¶

Response field¶

Every POST /v1/chat/completions response includes a usage field with prompt_tokens_details.cached_tokens:

{
  "usage": {
    "prompt_tokens": 512,
    "completion_tokens": 64,
    "prompt_tokens_details": {
      "cached_tokens": 480
    }
  }
}

cached_tokens is the number of prompt tokens whose KV state was inherited from a cached entry rather than recomputed. A value of 0 means a cold prefill (cache miss or feature disabled).

Prometheus metrics¶

All metrics are exposed at GET /metrics in Prometheus text format when the --metrics flag is set.

Hit rate formula:

hit_rate = mlxcel_prompt_cache_hits_total
         / (mlxcel_prompt_cache_hits_total + mlxcel_prompt_cache_misses_total)

Average tokens saved per hit:

avg_saved = mlxcel_prompt_cache_prefix_tokens_reused_total
          / mlxcel_prompt_cache_hits_total

Log output¶

The scheduler emits a DEBUG-level log line on each cache hit:

prompt-cache hit: adopted <matched>/<total> tokens  seq_id=<id>

and a DEBUG line on adopt failure:

prompt-cache adopt failed (<reason>); falling back to cold prefill

Troubleshooting¶

Cache miss when you expected a hit¶

1. template_sig drift: the chat template, its kwargs, or the tool schema changed between requests. Check that --chat-template, --chat-template-kwargs, and the tool list are identical. Even a whitespace difference in the template string produces a different digest.

2. session_key mismatch: if requests set different user values or different prompt_cache_key hints they land in different session buckets. Cross-session fallback only fires when the cross-session match is strictly longer than the in-session match.

3. min-prefix below threshold: the common prefix is shorter than --prompt-cache-min-prefix (default 32 tokens). Raise the threshold only if you know your shared prefix is longer; lower it if you want to capture shorter common prefixes.

4. First request of the session: no entry exists yet. The cache is populated only after a sequence finishes (healthy completion), so the very first request always cold-prefills.

5. Multimodal content changed: even if the text is identical, a different image or audio payload produces a different mm_digest and a cache miss.

Memory pressure and evictions¶

If mlxcel_prompt_cache_evictions_total{reason="lru"} grows fast:

• Raise --prompt-cache-capacity-bytes if you have headroom.
• Raise --prompt-cache-max-entries if entries are small.
• Lower --prompt-cache-ttl to remove stale entries sooner.

Session-key misuse across conversations¶

The user field is intended to be a stable end-user identifier. If the same user value is reused across completely unrelated conversations, all those requests land in the same session bucket. The cross-session matcher may pick up a prefix from an unrelated conversation that happens to share an early token sequence (e.g., a common system prompt). This is usually harmless but can cause surprising hits. To prevent cross-conversation collision, set a per-conversation prompt_cache_key in extra_body.

Tool-call non-determinism¶

Tool call arguments and responses are generated tokens and change on every run. The cache bucket stores the token stream up through the end of the last generation. For multi-turn conversations that include tool calls, the matching prefix will include the tool call schema (stable) but not the generated arguments or results (non-deterministic), so effective matched length may be lower than the full conversation history.

Limitations¶

The following are known limitations of the current implementation:

• Paged adopt and donate cover pool-backed families only: under --decode-storage-backend paged, adopt and donate share refcounted pool blocks through CachePool::adopt_paged and CachePool::detach_paged. A Dense entry is not adopted under the paged backend and a Paged entry is not adopted under the dense backend, because the KV shape cannot be installed into the active pool. Families with model-owned state use the snapshot path.

• Multimodal requests need a whole-entry match: a request with image or audio content adopts a cached entry only when the match covers the entire stored entry, so the prefilled suffix is the newly appended text. A partial (block-aligned) match is declined and the request prefills cold.

• No cross-process sharing: the cache lives in the server process heap. Multiple mlxcel-server processes do not share entries. Horizontal scale- out does not benefit from the cache across nodes.

• Single PromptCacheStore per process: all models loaded in the same process share one store. In practice each model has a unique model_id dimension so entries do not collide, but memory is shared.

See also: docs/architecture.md for the overall server architecture and where the prompt cache fits in the request processing pipeline.