# ctok font-only preset

The **ctok 4.8 (experimental)** preset uses a separate minimum-cost OpenType
backend. It is not a conversion to BPE. Compile with Inter, gg sans, or an uploaded
outline font, then install the resulting TTF or use its downloaded Vesktop theme.
No text preprocessing or tokenizer script is needed to display the compiled font.

## What is known about Claude

ctok is an unofficial reconstruction based on measured token counts. Its author
argues against ordinary byte-level BPE and presents minimum-piece segmentation as
an explanation of the measurements. This is evidence, not confirmation of
Anthropic's implementation. Counts do not identify a unique segmentation, and
ctok's tie-breaking need not match Claude's.

- [ctok source and accuracy qualifications](https://github.com/sanderland/ctok/tree/8c43ffd3c7be467622a540f025516e49ef78d8b1)
- [Author's research and non-BPE argument](https://tokencontributions.substack.com/p/on-the-biology-of-claudes-tokenizer)

The original compiler requires a ranked byte-BPE merge table. ctok supplies
measured vocabulary pieces, normalization/marker rules, byte fallback, and a
minimum-cost tiler. There is no equivalent merge table supplied by ctok; inventing
one would change the algorithm.

## What the font implements

1. OpenType substitutions add ctok word/case markers and normalize supported
   whitespace and controls. Presentation aliases preserve original case and v3
   curly-quote outlines while their semantic values follow ctok.
2. Contextual lookups indexed by their final character recognize vocabulary
   suffixes. Each result records all matching incoming edge lengths.
3. Boundaries that no vocabulary piece can cross divide the search graph into
   independent components. Breadth-first substitution passes find a shortest
   path inside each component; the decomposition preserves the minimum cost.
4. A reverse pass follows back-pointers. Sparse presentation glyphs divide each
   selected token into a total advance of 3 em; the TKSP axis adds 0–0.5 em per token.

This is real minimum-cost segmentation, not greedy longest matching. The tests
include `abcd` with pieces `ab` and `bcd`: the font selects `a + bcd`, avoiding the
more expensive greedy `ab + c + d` result.

## Scope and limitations

All three presets include the **full upstream vocabulary**: 48,332 pieces for
v3 and 14,819 for v4.7/v4.8. This removes both the former 6,000-piece cutoff and
the later Latin-only projection. Semantic vocabulary coverage is distinct from
visible coverage: a character still needs an outline in the chosen source font.
Unsupported visible characters are left out of the cmap so ordinary font
fallback remains visible rather than silently drawing blanks.

Singleton NFC aliases (including Kelvin/Angstrom signs and Greek variants) are
normalized explicitly. An exhaustive audit checks all 2,103 printable Inter
codepoints against each ctok version; this does not prove arbitrary combining
sequences normalize correctly. See [the repertoire audit](ctok-4.8-unicode-repertoire.json).

The compiler extends the repertoire to source-font characters using pinned ctok
Unicode classes and byte fallback costs. Combining marks retain token advance;
stripped controls and folded Unicode spaces follow ctok, including nonempty
inputs that normalize to empty. The normalizer also handles framing seams,
CRLF, title case and v3 long all-capital words. v3 quote folding keeps the original
curly outlines. Message overhead is excluded (7/11/6 tokens respectively).

Search supports **256 steps per independent vocabulary-connected component**,
including its frame markers. Paragraphs can exceed 256 total tokens when exact
graph cuts separate their components. If a connected component exceeds this
budget, a visible `[limit]` marker appears alongside unconverted text; the result
must not be treated as an equal-width tokenization. A truly empty input cannot
emit the frame token required by v3/v4.7 because there is no input glyph to
substitute.

Unicode normalization performed by the shaping engine, joining scripts, font
fallback, formatting and browser line wrapping can create different streams or
separately shaped runs. Literal reserved U+FDD0–U+FDEF inputs also remain unsupported.
The compiler copies outlines rather than the source font’s entire shaping program;
contextual joining and emoji/other source ligatures are not generally preserved.
Full vocabulary coverage does not remove those limits.
This is not a billing counter, and matching ctok does not establish Claude's
segmentation or tie-breaking.

The indexed lookups and graph decomposition reduced one browser stress case
(1,300 characters at 24 px in a 320 px-wide wrapped box) from about 50.8 seconds
to 5.81 seconds with the final extended repertoire. That remains too slow for unrestricted long messages; wrapping
can shape the same text repeatedly. See [browser timings](ctok-browser-performance-fixes.json).

See [current preset details](tokenizer-presets.md) and the
[complete corpus and adversarial regression](ctok-backend-fixes.json).

## Earlier verification (before the expanded repertoire)

The standalone TTF passed 705 seeded checks against an independent dynamic
programming oracle and ctok's content counts: vocabulary words, word sequences,
random lowercase strings, and directed examples. This is a targeted ASCII test,
not a general-language accuracy benchmark. See [the verification report](ctok-verification.json).

The browser compiled the font locally and rendered `hello, world` at 240 px with
20 px text: four tokens × 3 em. Tests also cover a greedy counterexample, spacing,
standalone GSUB tables, preset switching, and the existing BPE backend.

Source pinned at `8c43ffd3c7be467622a540f025516e49ef78d8b1`.
ctok's [MIT license](../licenses/ctok-MIT.txt) is bundled with the website.

### Broader and live-model audit

A subsequent audit matched ctok on all 539 in-scope cases out of 554 total;
15 cases exercised unsupported Unicode, tab/newline input or excessive run length.
Ten live ASCII probes matched Sonnet 5's server-reported input-count differences.
ctok also matched three live Unicode probes. Three baseline controls were stable.
See [methodology, limitations and raw results](ctok-live-validation.md).