# Tokenizer presets

ctok v3, v4.7 and v4.8 use a font-only minimum-cost backend with the **full
upstream vocabulary**: 48,332 pieces for v3 and 14,819 for v4.7/v4.8. Indexed
contextual edges and sparse presentation glyphs remove the earlier vocabulary
projection without exceeding the glyph limit. Visible coverage still depends on
the uploaded source font; missing outlines retain ordinary font fallback.

The normalizer handles title case, v3 long all-capital words, contractions,
leading spaces, CRLF, stripped controls, folded Unicode spaces and trailing
newline framing. Source-font characters gain pinned ctok Unicode classes and
byte fallback costs. v3 curly quotes retain their original outlines. Message
frame overhead is excluded (7/11/6 tokens). Nonempty normalized-empty inputs are
handled; an actual empty input cannot display the v3/v4.7 frame token.

Exact graph cuts allow more than 256 total tokens per paragraph. The remaining
budget is 256 search steps per vocabulary-connected component, including frame
markers. Overflow displays `[limit]` and preserves unconverted text rather than
pretending the result has equal token widths. Browser normalization, joining
scripts, wrapping and formatting remain limitations, and long wrapped messages
can still be slow. ctok approximates Claude; matching it does not establish
Claude's segmentation or tie-breaking. See [ctok scope and verification](ctok-compatibility.md).

OpenAI presets include all mergeable vocabulary entries, including 199,998 for
o200k_base. Their regex boundaries are now implemented inside the font with a
backward suffix-reachability pass and a forward matching pass before BPE.
The compiler still reconstructs canonical merge paths; special-token spellings
are ordinary text. Source-font coverage and browser shaping boundaries remain
limitations. Generic uploaded raw-BPE presets do not automatically gain this
OpenAI-specific frontend.

See [the actual-font Gutenberg audit](gutenberg-font-fixes.json) and
[the earlier baseline](gutenberg-audit.md). The baseline describes the older
compiler, before these normalization and regex fixes.

Sources: [ctok](https://github.com/sanderland/ctok) and
[OpenAI tiktoken](https://github.com/openai/tiktoken). Source revisions and versions
are recorded in each preset. Regenerate using `prepare-ctok.py` and
`prepare-openai.py`. The website compiles locally; uploads stay in the browser.


Kimi K3 now uses the pinned original regex and byte-rank canonical reconstruction,
checked against Moonshot's vocabulary. Qwen 3.6 preserves its original regex,
all causal merge pairs, and bounded font-only NFC. Both pass all 5,493 Gutenberg
chunks and all 4,000 covered prose/code/whitespace/punctuation fuzz cases. Qwen's
NFC bound is eight consecutive nonstarters after decomposition; longer sequences
display `[NFC limit]`. Missing normalized outlines are separately reported.

Additional adversarial audits: [browser rendering](browser-domain-findings.md),
[ctok](ctok-domain-audit.md), [OpenAI BPE](bpe-domain-audit.md),
[DeepSeek](deepseek-pathway-repaired.md), [Qwen](qwen-pathway-repaired.md),
[Kimi](kimi-pathway-audit.md), [GLM](glm-pathway-audit.md), and
[original Llama 3](llama3-pathway-audit.md). The pathway suites use the complete Gutenberg
corpus plus 5,000 generated strings per tokenizer, each exactly 32 reference
tokens. Missing glyph coverage is reported separately from algorithm differences.

Confirmed remaining limitations include browser canonical normalization, run
boundaries from formatting, unsupported-character fallback, and the ctok search
budget. ctok can be very slow on long runs. GLM's currently shipped raw preset
still omits parts of the original tokenizer; its replacement priority backend
is under test. Original Llama 3 has the same scheduling/whole-piece challenges.
Matching an adapted preset is weaker than matching the full reference.
See the [active discrepancy tracker](discrepancies.md) for repair status.