Language Bias¶

Table of Contents¶

• What Is Language Bias?
• Quickstart
• Flag Reference
• YAML Config Schema
• Inclusion Policy
• Exception Flags
• Cache
• Observability
• Limitations
• Use-Case Examples
• Troubleshooting

What Is Language Bias?¶

Language bias is mlxcel's mechanism for output-distribution surgery at the vocabulary level. Rather than applying post-processing filters or steering at the activation level, it operates directly on the logits produced by the language model during token sampling: before any temperature scaling or repetition penalty, an additive bias is applied to the raw logits of every token that belongs to a specified language. Tokens biased to -inf are permanently removed from the probability distribution (their softmax weight becomes exactly zero). Tokens given a positive bias are made more likely. Because this happens inside the sampling kernel itself, it is zero-overhead when no policy is active and incurs only a vector-add cost when one is configured.

The preset layer maps human-readable language codes to vocabulary token sets by scanning the tokenizer's full vocabulary at first load and classifying each token by its Unicode script properties. The result is cached to disk, so later starts skip the scan. Two classification policies, Conservative and Strict, let you tune whether mixed-script tokens (for example, Korean text that includes Chinese characters for Sino-Korean vocabulary) follow the broader or narrower definition of each language.

Quickstart¶

Suppress one language and promote another (recommended)¶

Tell the model both which languages to avoid and which to prefer. Suppression alone pushes the distribution toward whatever is left (digits, whitespace, punctuation) and rarely produces a useful response; pairing it with a positive bias on the target language gives the model somewhere to go.

mlxcel generate \
  --model ./models/Qwen2.5-7B-Instruct-4bit \
  --prompt "日本の首都は" \
  --lang-bias ja=-inf,zh=-inf,ko=+5 \
  --max-tokens 100

Verified on qwen2.5-0.5b-bf16: without the bias the model continues in Japanese (日本の首都は東京都です); with the bias above it switches to coherent Korean (답변하신 내용은 한국의 수도 서울입니다. 서울은 대한민국의 수도로 인해 세계적으로 유명한 대학과 연구...). A promote bias of +5 to +10 is a safe starting range. Values around +20 or higher tend to cause repetition as the promoted tokens dominate the distribution too aggressively.

Suppress only (guardrail use)¶

Appropriate when you need to guarantee that a particular language never appears in output and you accept incoherent output as the fallback:

mlxcel generate \
  --model ./models/Qwen2.5-7B-Instruct-4bit \
  --prompt "今日のニュースを教えてください。" \
  --lang-bias ja=-inf,zh=-inf \
  --max-tokens 100

Expect numeric- or punctuation-heavy output when the prompt's natural continuation was in a suppressed language. The model still samples, just from the non-suppressed tail.

Server with a global bias¶

mlxcel serve \
  --model ./models/Qwen2.5-7B-Instruct-4bit \
  --lang-bias ja=-inf,zh=-inf,ko=+5 \
  --port 8080

Flag Reference¶

--lang-bias <entries>¶

Apply additive logit biases to language token sets.

Syntax:

--lang-bias <entry>[,<entry>]*
<entry>  = <lang_code>=<bias>
<bias>   = -inf | +inf | inf | <float>
<lang_code> = ja | zh | ko | en | ru | ar | th | hi | he | el

Default: none (no bias applied).

Examples:

--lang-bias ja=-inf,zh=-inf
--lang-bias ko=+5.0
--lang-bias en=+3,ja=-inf,zh=-5.5

Duplicate language codes in the same entry set produce a parse error. Bias values are parsed as 32-bit floats; -inf and +inf are accepted as literals.

--lang-bias-config <path>¶

Load the language bias configuration from a YAML file instead of the inline flag. See the YAML Config Schema section for the full format.

Default: none.

Example:

mlxcel generate \
  --model ./models/Qwen2.5-7B-Instruct-4bit \
  --prompt "Hello" \
  --lang-bias-config ./lang-policy.yaml

The YAML file may specify the policy, bias entries, and exception settings together. If both --lang-bias and --lang-bias-config are supplied, they are merged; inline --lang-bias entries take precedence over entries for the same language in the YAML file.

--lang-bias-policy {conservative|strict}¶

Set the token inclusion policy for all language sets.

Default: conservative.

Values:

Example:

mlxcel generate \
  --model ./models/Qwen2.5-7B-Instruct-4bit \
  --prompt "한국어 시험 문제" \
  --lang-bias ko=-inf \
  --lang-bias-policy strict

--lang-bias-include-special¶

Include the tokenizer's special tokens (BOS, EOS, PAD, UNK, template delimiters) in script classification. By default these are excluded from every language set so that suppressing a language never blocks end-of-sequence generation.

Default: off (special tokens are excluded).

--lang-bias-include-numeric¶

Include purely numeric tokens in script classification. By default 0-9 sequences and Unicode numeric characters are excluded.

Default: off.

--lang-bias-include-punctuation¶

Include purely punctuation tokens in script classification.

Default: off.

--lang-bias-include-byte-fragments¶

Include byte-fragment tokens in script classification via UTF-8 start-byte analysis.

Byte-level BPE tokenizers (Qwen, GPT-2, LLaMA, Mistral) represent less-common CJK characters as sequences of individual byte tokens. Each byte decodes to U+FFFD on its own and classifies as Other, so the default classifier does not let ja=-inf or zh=-inf touch those fragments. They then reassemble into the target character during generation.

Enabling this flag runs a second vocab-scan pass that reverse-maps each byte-fragment token back to its raw byte and tags it with a likely script based on the UTF-8 start-byte range (0xE3 → Hiragana, 0xE4-0xE9 → Han, 0xEA-0xED → Hangul, 0xF0-0xF4 → Han for supplementary planes). Continuation bytes (0x80-0xBF) stay Other because they are ambiguous across scripts.

Start-byte classification is an approximation. The 0xE4-0xE9 range catches most CJK Unified Ideographs but also covers parts of Latin Extended Additional; operators who care about precision should watch mlxcel_lang_bias_byte_fragment_suppressions_total on /metrics to see how many suppressions are byte-fragment-driven and back out if over-suppression becomes visible in their output.

Default: off. When off, byte-fragment tokens remain excluded from script classification.

Example:

mlxcel generate \
  --model ./models/Qwen2.5-7B-Instruct-4bit \
  --prompt "日本の首都は" \
  --lang-bias ja=-inf,zh=-inf,ko=+5 \
  --lang-bias-include-byte-fragments \
  --max-tokens 100

With this flag enabled, the Qwen2.5 tokenization [74577, 112] that produces 年 in output (documented under Limitations below) now has its leading byte-fragment tagged as Han and caught by zh=-inf.

--lang-bias-rebuild-cache¶

Force a full rebuild of the TokenLanguageIndex cache for the current model's tokenizer. The existing cache file is discarded and a new one is written. This is rarely needed; the cache is normally invalidated automatically when the vocabulary changes.

Default: off.

LLAMA_ARG_LANG_BIAS (environment variable)¶

The same syntax as --lang-bias, read from the environment. Used when running mlxcel-server through a process manager or container orchestration system that sets LLAMA_ARG_* variables. The command-line flag always takes precedence over this environment variable.

Example:

export LLAMA_ARG_LANG_BIAS="ja=-inf,zh=-inf"
mlxcel serve --model ./models/Qwen2.5-7B-Instruct-4bit

LLAMA_ARG_LANG_BIAS_INCLUDE_BYTE_FRAGMENTS (environment variable)¶

Enables the byte-fragment classifier from the environment. Accepts true/false/1/0/yes/no/on/off (case-insensitive). Unparseable values are logged and ignored; no parse error is raised. --lang-bias-include-byte-fragments on the command line takes precedence.

Example:

export LLAMA_ARG_LANG_BIAS="ja=-inf,zh=-inf,ko=+5"
export LLAMA_ARG_LANG_BIAS_INCLUDE_BYTE_FRAGMENTS=true
mlxcel serve --model ./models/Qwen2.5-7B-Instruct-4bit

YAML Config Schema¶

The YAML configuration file accepted by --lang-bias-config has three top-level keys:

# Inclusion policy: conservative (default) or strict.
policy: conservative

# Per-language logit biases. Supports -inf, +inf, and any float.
bias:
  ja: -inf
  zh: -10.0
  ko: +5.0

# Exception settings. All default to false.
exceptions:
  include_special: false
  include_numeric: false
  include_punctuation: false
  include_byte_fragments: false

All keys are optional. An empty file is valid and applies no bias.

Inclusion Policy¶

The inclusion policy controls how tokens that contain characters from multiple scripts are assigned to language sets.

Conservative vs Strict Matrix¶

For most languages, Conservative and Strict are identical. Korean is the notable exception.

Worked Example: Korean¶

Modern Korean text mixes Hangul syllables with Sino-Korean vocabulary written in Chinese characters (Han script). This creates a meaningful policy difference:

Conservative (--lang-bias-policy conservative, the default):

• Korean language set = tokens containing Hangul or Han characters.
• --lang-bias ko=-inf suppresses both Hangul tokens and standalone Chinese-character tokens.
• This gives the most complete suppression of Korean-language output but also removes Han characters that are shared with Japanese and Chinese vocabulary.

Strict (--lang-bias-policy strict):

• Korean language set = tokens whose all characters are Hangul.
• --lang-bias ko=-inf suppresses only pure-Hangul tokens. Han-character tokens are unaffected.
• Japanese and Chinese vocabulary that uses Han characters remains available even when Korean is suppressed.

Choosing between the two:

• Use Conservative when you want to prevent Korean-language output entirely and do not need Han characters in output.
• Use Strict when you need Korean suppression that preserves the ability to output CJK shared characters (for example, when the model must still output Japanese or Chinese alongside or instead of Korean).

Exception Flags¶

Three categories of tokens are excluded from every language set by default regardless of the inclusion policy:

Whitespace-only tokens are always excluded and cannot be overridden.

When to use the override flags:

• --lang-bias-include-special: rarely needed. Only relevant if you are trying to prevent the model from generating end-of-text or pad tokens explicitly. Enabling this when suppressing a language can cause generation to stall if the model cannot produce EOS.
• --lang-bias-include-numeric: useful when you want to suppress all tokens associated with a script including number sequences in that script (for example, Arabic-Indic numerals in ar suppression). Has no effect for scripts that do not have distinct numeric characters.
• --lang-bias-include-punctuation: useful when the target script has unique punctuation characters (for example, Japanese corner brackets 「」 or Korean quotation marks). Without this flag, those punctuation tokens remain available even when the language is suppressed.

Cache¶

Location¶

${MLXCEL_CACHE_DIR:-$HOME/.cache/mlxcel}/tokenizer-scripts/<vocab_hash>.bin

An empty MLXCEL_CACHE_DIR counts as unset.

<vocab_hash> is the first 16 hex characters of the SHA-256 digest of the raw tokenizer.json bytes. The file is postcard-serialized and contains the full TokenLanguageIndex for the vocabulary.

Automatic Invalidation¶

The cache is rebuilt automatically when:

• The cache file does not exist (first run with this tokenizer).
• The cache file's schema version does not match the current mlxcel version.
• The cache file cannot be read (truncated or corrupt). The old file is renamed to <path>.broken.<epoch>.bak before the rebuild.

Manual Rebuild¶

mlxcel generate \
  --model ./models/Qwen2.5-7B-Instruct-4bit \
  --lang-bias-rebuild-cache \
  --prompt "Hello" \
  --max-tokens 1

Or with the server:

mlxcel serve \
  --model ./models/Qwen2.5-7B-Instruct-4bit \
  --lang-bias-rebuild-cache

The --lang-bias-rebuild-cache flag does not require a bias to be set; it rebuilds and caches the index for any subsequent run.

Observability¶

When using mlxcel-server, the /metrics endpoint (Prometheus format) exposes three counters for monitoring language bias activity:

Example:

curl -s http://localhost:8080/metrics | grep lang_bias

Expected output when bias is active:

# HELP mlxcel_lang_bias_applied_total Sampling steps with a non-empty language bias map applied
# TYPE mlxcel_lang_bias_applied_total counter
mlxcel_lang_bias_applied_total 1482
# HELP mlxcel_lang_bias_tokens_suppressed_total Sampling steps where the top-1 candidate was suppressed by -inf bias
# TYPE mlxcel_lang_bias_tokens_suppressed_total counter
mlxcel_lang_bias_tokens_suppressed_total 374
# HELP mlxcel_lang_bias_byte_fragment_suppressions_total Suppressions where the neg-inf top-1 token was classified via UTF-8 start-byte
# TYPE mlxcel_lang_bias_byte_fragment_suppressions_total counter
mlxcel_lang_bias_byte_fragment_suppressions_total 41

Tracing fields (visible with RUST_LOG=debug or a distributed tracing backend):

• lang_bias.entries: number of active bias entries
• lang_bias.byte_fragment_entries: number of byte-fragment entries (emitted only when non-zero)
• lang_bias.languages: comma-separated list of biased language codes
• lang_bias.policy: conservative or strict

Limitations¶

Single global policy per serving instance. The language bias configuration applies to all requests handled by one mlxcel-server instance. There is no per-request override in the OpenAI-compatible API.

10-language coverage. Language bias currently supports ja, zh, ko, en, ru, ar, th, hi, he, and el. Adding new languages requires a tokenizer validation pass.

Script::Other tokens are never classified. Tokens whose characters fall into Unicode scripts not covered by the 10-language set (for example, Georgian, Armenian, or mathematical symbols) are classified as Script::Other and do not enter any language set. They are never affected by language bias.

Per-sequence batch differentiation deferred. When multiple sequences are processed in a batch, all sequences use the same language bias policy. Serving different language policies to different concurrent requests within a single serving instance requires per-request override support.

Byte-fragment CJK escape on byte-level BPE tokenizers. Byte-level BPE tokenizers (Qwen, GPT-2, LLaMA, Mistral) represent less common CJK characters as a sequence of individual byte tokens rather than a single merged whole-character token. In the default classifier each byte decodes to U+FFFD on its own and classifies as Other, so ja=-inf and zh=-inf do not touch it; the fragments reassemble into the target character at generation time and appear in output despite the suppression.

The opt-in --lang-bias-include-byte-fragments flag (and the matching LLAMA_ARG_LANG_BIAS_INCLUDE_BYTE_FRAGMENTS env var) closes most of this gap. When the flag is on, the vocab scan reverse-maps each byte-fragment token back to its raw byte and tags it with a likely script via UTF-8 start-byte analysis. The concrete Qwen2.5-0.5b check: 年 standalone is a single token (7948, decodes to 年, classified as Han, bias applied in both modes); the same character preceded by a space tokenizes as two byte fragments (74577, 112). With the flag off, both fragments stay Other and bypass zh=-inf. With the flag on, the leading fragment 74577 is tagged as Han via its start-byte range and the suppression applies.

The byte-fragment pass is not hermetic. Continuation bytes (0x80-0xBF) remain ambiguous across scripts and stay Other, so a CJK character whose start byte is suppressed but whose continuation bytes slip through other language sets can still surface in degenerate cases. Start-byte classification is also approximate. The 0xE4-0xE9 range catches most CJK Unified Ideographs but also covers parts of Latin Extended Additional, so enabling the flag may over-suppress a small population of unrelated tokens. Operators should watch mlxcel_lang_bias_byte_fragment_suppressions_total to see how much of their suppression comes from the byte-fragment path and back the flag out if the volume looks disproportionate. Treat the combined filter as a stronger preference, not a hermetic block.

Use-Case Examples¶

Example 1: Korean-Only Serving (Conservative)¶

A Korean-language customer support service that must not produce Japanese or Chinese output:

mlxcel serve \
  --model ./models/Qwen2.5-7B-Instruct-4bit \
  --lang-bias ja=-inf,zh=-inf \
  --lang-bias-policy conservative \
  --port 8080

With Conservative policy, Hiragana, Katakana, and Han-script tokens are all removed from the distribution. The model's Korean Hangul vocabulary and Latin-script tokens (for proper nouns and technical terms) remain.

Example 2: Korean-Only Serving Preserving Hanja¶

A Korean-language service where the model may need to output Hanja (Chinese characters used in Korean contexts) but must not output Japanese kana:

mlxcel serve \
  --model ./models/Qwen2.5-7B-Instruct-4bit \
  --lang-bias ja=-inf \
  --lang-bias-policy strict \
  --port 8080

Under Strict policy, ja=-inf suppresses only tokens whose characters are all Hiragana or Katakana (or mixed kana+Han). Tokens that contain only Han characters are not in the ja strict set and remain available. This preserves Sino-Korean vocabulary while blocking Japanese kana output.

Alternatively, to suppress Korean Hangul itself while keeping Han available (for testing or migration scenarios):

mlxcel generate \
  --model ./models/Qwen2.5-7B-Instruct-4bit \
  --prompt "한국어 시험 문제" \
  --lang-bias ko=-inf \
  --lang-bias-policy strict \
  --max-tokens 50

Example 3: English Exam Serving (Suppress Non-Latin)¶

An English-language examination platform that must ensure all outputs use only Latin-script characters:

mlxcel serve \
  --model ./models/Qwen2.5-7B-Instruct-4bit \
  --lang-bias ja=-inf,zh=-inf,ko=-inf,ru=-inf,ar=-inf,th=-inf,hi=-inf,he=-inf,el=-inf \
  --lang-bias-policy conservative \
  --port 8080

This removes all nine non-Latin language sets from the distribution, leaving Latin-script output, numeric tokens, punctuation, and special tokens.

Using a YAML config file for the same setup:

# exam-english-only.yaml
policy: conservative
bias:
  ja: -inf
  zh: -inf
  ko: -inf
  ru: -inf
  ar: -inf
  th: -inf
  hi: -inf
  he: -inf
  el: -inf

mlxcel serve \
  --model ./models/Qwen2.5-7B-Instruct-4bit \
  --lang-bias-config ./exam-english-only.yaml \
  --port 8080

Troubleshooting¶

Bias has no effect¶

The model output appears unchanged after adding --lang-bias.

1. Confirm the flag spelling. Run mlxcel generate --help and verify the exact flag name. A typo silently passes through as an unrecognized flag in some shell configurations.
2. Check that the language you are suppressing actually appears in this model's vocabulary. Not all models have equal multilingual coverage. A model trained primarily on English may have very few Japanese tokens; suppressing them has no visible effect on outputs that were already English.
3. Check the metrics counter. When using mlxcel-server, curl /metrics | grep lang_bias_applied_total should show a non-zero count if bias is being applied each step. A zero count indicates the bias map is empty (the tokenizer scan produced no tokens for the specified language).
4. Rebuild the cache. Run with --lang-bias-rebuild-cache once to force a fresh vocabulary scan.
5. A few suppressed-language characters still appear despite a non-zero applied counter. This is the byte-fragment CJK escape described under Limitations. By default, byte-fragment tokens are classified as Other, which bypasses the filter; the fragments reassemble at generation. Two options:
6. Enable --lang-bias-include-byte-fragments (or set LLAMA_ARG_LANG_BIAS_INCLUDE_BYTE_FRAGMENTS=true). This tags byte-fragment tokens via UTF-8 start-byte analysis and catches most of the escape. Monitor mlxcel_lang_bias_byte_fragment_suppressions_total on /metrics to verify the flag is actually catching suppressions, and back out if over-suppression becomes visible (for example, unexpected Latin Extended tokens disappearing).
7. Pair the negative bias with a promote on the target language (e.g. ja=-inf,zh=-inf,ko=+5) so the model has somewhere to go that does not pass through the escape path. This helps even without the byte-fragment flag, because the promote pulls the distribution toward tokens that do not participate in the escape.

Output quality degraded¶

The model produces incoherent output, repetitions, or very short responses after adding --lang-bias.

1. Reduce the magnitude. Change -inf to a large negative float such as -10.0. Hard suppression (-inf) removes tokens from the distribution entirely; if the model relies on those tokens for grammatical structure, quality drops sharply.
2. Switch to Strict policy. Conservative policy may be suppressing more tokens than expected (for example, Han-script tokens that the model also uses for structural purposes). Adding --lang-bias-policy strict narrows the suppressed set.
3. Re-enable punctuation. Some punctuation tokens are script-associated (for example, Japanese full-width punctuation). Adding --lang-bias-include-punctuation puts them back in the available set.
4. Check for unintended EOS suppression. If the suppressed language set contains the EOS token (unlikely but possible with custom vocabularies), add --lang-bias-include-special to verify, then check whether EOS is in the suppressed set.

Cache rebuild failure¶

mlxcel reports an error reading or writing the cache file.

1. Check disk space. The cache directory is ${MLXCEL_CACHE_DIR:-$HOME/.cache/mlxcel}/tokenizer-scripts/. A large vocabulary (150k+ tokens) may require several megabytes.
2. Delete the corrupt file manually. Find and remove the file at the path shown in the error message:

   ls ~/.cache/mlxcel/tokenizer-scripts/
   trash-put ~/.cache/mlxcel/tokenizer-scripts/<vocab_hash>.bin
   

   Then rerun with --lang-bias-rebuild-cache.
3. Set MLXCEL_CACHE_DIR to a writable location if the default cache path is on a read-only filesystem:

   export MLXCEL_CACHE_DIR=/tmp/mlxcel-cache
   mlxcel generate --model ./models/... --lang-bias ko=-inf --prompt "Hello" --max-tokens 5