TurboQuant KV Cache Compression¶

TurboQuant is a KV cache compression mode for long-context inference. It lets mlxcel store part of the attention cache in a compact quantized form, reducing memory use while keeping the model weights unchanged.

In practical terms, TurboQuant is for the moment when the model itself fits, but the memory used to remember a long conversation or several simultaneous requests does not. It makes that "remembered context" smaller.

Use TurboQuant when the model fits in memory but the context window, batch size, or number of concurrent requests makes the KV cache too large. Keep the default fp16 cache when memory is not constrained, when you are comparing against a baseline, or when the model family is not listed in the supported configuration table below.

The implementation follows the TurboQuant paper (Zandieh et al., ICLR 2026, arXiv:2504.19874) and the PolarQuant-only refinement validated by the independent TurboQuant+ project.

Table of Contents¶

• Why the KV Cache Matters
• What TurboQuant Changes
• Which Mode to Start With
• Cache Modes
• CLI Flags
• Examples
• Tuning
• Supported Configurations
• Known Limitations

Why the KV Cache Matters¶

During autoregressive generation, the model does not recompute every previous token from scratch. Each transformer layer stores two tensors for every token that has already been processed:

• K, or keys: used to compute attention scores for the next token.
• V, or values: mixed together according to those attention scores to form the next hidden state.

This stored state is the KV cache. It makes decoding practical, but it grows linearly with context length, layer count, the number of KV heads, the size of each head vector, and the number of active sequences. Here, an active sequence means one request or batch entry being decoded. At short context the cache may be small compared with model weights. At long context or high concurrency, it can become one of the main memory costs.

Think of the KV cache as the model's working notes for the current request. A longer prompt, a larger context window, or more simultaneous users means more notes. TurboQuant compresses those notes so a longer session can stay in memory.

What TurboQuant Changes¶

TurboQuant changes how mlxcel stores the KV cache. It does not change the model weights, the tokenizer, or the prompt. It does two things:

1. Put the cache vector into a shape that is easier to compress.
2. Store it with fewer bits per value.
3. Expand it back when attention needs to read it.

The first step uses a Walsh-Hadamard transform, and the compact storage uses a PolarQuant codebook. You do not need to tune those pieces directly; choose the cache mode and boundary setting instead.

The default TurboQuant mode in mlxcel is fp16+turbo4. It keeps K in FP16 and compresses V to 4-bit TurboQuant storage.

This default is intentionally asymmetric. K is used to decide where the model attends. If K is distorted, the model may attend to the wrong tokens. V is the content mixed after attention has chosen where to look, so V can usually be compressed with less risk. That is why fp16+turbo4 is the recommended starting point for dense Q4KM models.

Boundary-V protection keeps the first and last few transformer layers in FP16 V. Those layers are often more sensitive to compression, so this policy trades some memory savings for better output quality.

Which Mode to Start With¶

For a first deployment, prefer fp16+turbo4 with the default boundary setting. Move to the more aggressive modes only after checking output quality on your own prompt distribution.

Cache Modes¶

The simple memory formula is (K_bits + V_bits) / (2 * 16) plus small per-block norm sidecars. Boundary-V protection keeps selected V layers in FP16, so the actual memory footprint is higher than the table value for Turbo modes.

WHT in the table means Walsh-Hadamard transform. It is the rotation step that makes the cache vectors easier to store with a small codebook.

CLI Flags¶

mlxcel generate, mlxcel serve, and mlxcel-server share the same TurboQuant flags.

When a split flag and --kv-cache-mode are both supplied, the split flags take precedence and mlxcel logs a warning. If only one split flag is supplied, the other side defaults to fp16.

Supported K/V split combinations¶

Other combinations are rejected at startup with a message listing the supported pairs. Symmetric Turbo3 (turbo3 / turbo3) is not exposed.

The split flags also accept f16, llama.cpp's spelling of unquantized FP16 storage, as fp16. Every other GGML cache type (q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1, f32, bf16) is rejected with an explanation, because none of them is numerically equivalent to an mlxcel mode. int8 here is per-token absmax, and the Turbo modes are PolarQuant with a Walsh-Hadamard rotation. fp16+turbo4-delegated is accepted as an alias of turbo4-delegated.

Examples¶

Offline generation¶

# Recommended TurboQuant default: FP16 K, Turbo4 V.
mlxcel generate -m models/<model> --kv-cache-mode fp16+turbo4 -p "Hello" -n 200

# Symmetric Turbo4. Use only for allowlisted model families.
mlxcel generate -m models/<model> --kv-cache-mode turbo4 -p "Hello" -n 200

# Delegated cache path.
mlxcel generate -m models/<model> --kv-cache-mode turbo4-delegated -p "Hello" -n 200

Server¶

# Unified mode flag.
mlxcel-server -m models/<model> --kv-cache-mode fp16+turbo4

# llama-server-compatible split flags.
mlxcel-server -m models/<model> --cache-type-k fp16 --cache-type-v turbo4

# Equivalent environment variables.
LLAMA_ARG_CACHE_TYPE_K=fp16 LLAMA_ARG_CACHE_TYPE_V=turbo4 mlxcel-server -m models/<model>

Tuning¶

Boundary-V layer protection¶

When a Turbo mode is active, the first and last --turbo-boundary-v transformer layers keep FP16 V caches. The default is 2, matching the LA-V7 policy used by the TurboQuant+ validation work.

mlxcel generate --kv-cache-mode fp16+turbo4 --turbo-boundary-v 2 \
    -m models/<model> -p "Hello" -n 50

# Equivalent environment variable.
MLXCEL_KV_BOUNDARY_V_LAYERS=4 mlxcel generate --kv-cache-mode fp16+turbo4 ...

Special values:

• 0: disable boundary protection.
• 2: default, suitable for typical 28- to 64-layer models.
• Values larger than n_layers / 2 are clamped. For example, a 4-layer model with --turbo-boundary-v 8 keeps all 4 layers at FP16 and does not error.

Sparse-V dequant threshold¶

For attention positions whose weight is below the threshold, mlxcel can skip V dequantization because the contribution to the output is negligible.

MLXCEL_SPARSE_V_THRESHOLD=1e-5 mlxcel generate --kv-cache-mode fp16+turbo4 ...

The default threshold is 1e-6. On macOS, the fused Metal kernel is enabled when the threshold is non-zero. Set MLXCEL_SPARSE_V_KERNEL=0 to compare against the graph reference path.

Delegated cache variants¶

The delegated cache keeps recent tokens in FP16 and stores older V tokens in Turbo4 form. In other words, the cache is split into a recent part that stays easy to read and an older part that is stored compactly. The variables below are advanced controls for experiments and performance comparisons.

Supported Configurations¶

Current symmetric turbo4 allowlist entries are qwen3_5, qwen3_5_moe, and qwen3_next. Q8 and 70B+ Q4KM models are candidates for future allowlist entries, but they should not be documented as generally enabled until the runtime can identify the weight tier and the specific family has been tested.

Dense Q4KM models outside the allowlist can show large PPL regressions with symmetric turbo4 because K-side quantization affects attention scores. Use fp16+turbo4 for those models.

Known Limitations¶

Head-dimension constraint¶

TurboQuant requires a power-of-two head dimension, such as 64, 128, or 256. Models with head dimensions such as 80, 96, or 192 cannot use TurboQuant without padding and are rejected during cache initialization.

Symmetric Turbo3¶

Symmetric turbo3 is not exposed. --kv-cache-mode turbo3 and --cache-type-v turbo3 resolve to the asymmetric fp16+turbo3 mode.

Paged KV scheduler¶

--kv-cache-mode <turbo4-variant> --decode-storage-backend paged is wired through the server dispatch path. End-to-end validation on Apple Silicon for decode throughput and concurrent-prefix coherence is still in progress.

M5 Max delegated decode speed¶

The default compressed turbo4-delegated path on M5 Max has not consistently met the 0.97x FP16 decode-speed target. The MLXCEL_TURBO4_DELEGATED_FP16_FAST_PATH=1 variant can be used for speed experiments, but it keeps a full FP16 V working set and therefore changes the memory profile.

Pre-M3 hardware¶

turbo3 can be bandwidth-bound on M1 and M2 GPU microarchitectures. Prefer fp16+turbo4 for decode-bound workloads on M1/M2.

VLM coverage¶

Current measurements cover Qwen2-VL 2B as a stand-in for Qwen 2.5-VL 3B, and Aya-Vision 8B at contexts up to 4K. Other VLM families and multi-image prompts above 8K tokens still need separate validation.