# Font compiler discrepancies and limitations

This is the working checklist for the font-only compiler. A fix is tied to its
recorded reproduction and test scope; it is not a claim of universal tokenizer
or browser equivalence. Tests compare reference IDs, counts, and advances
separately. No live Claude calls are used.

## Compiler discrepancies

| ID | Issue | Current status and evidence |
| --- | --- | --- |
| B1 | **GLM loses merge paths, regex aboundaries and whole-piece shortcuts.** `oul`, `_ca` and ` rd` split incorrectly. Preserving all pairs alone still misses newly enabled earlier-priority merges (` руками`) and whole-piece ` 参考`. | **In progress.** All 321,649 pairs now fit a 64,660-glyph font, but ordinary ordered passes are insufficient. A bounded exact-priority GSUB prototype passes random/backstep/leftmost tests; full-vocabulary scaling is running. [Factorization](glm-pathway-factorization.md), [rank audit](rank-canonical-audit.json). |
| B2 | **Kimi's first-available merge-path conversion is wrong.** `hello` became `he + l + lo`. | **Fixed for tested ordinary text.** Byte-rank canonical reconstruction plus the original regex now passes all 5,493 Gutenberg chunks and all 4,000 non-Unicode fuzz cases, including token advances. The 1,000 Unicode cases include 999 unsupported by Inter and one renderer-normalization discrepancy. Corrected preset enabled in UI. [Results](kimi-pathway-audit.md). |
| B3 | **Qwen, GLM, Kimi and Llama omit their original regex.** Qwen had 304 Gutenberg mismatches attributable to this. GLM/Llama use older cl100k-style whitespace branches. | **Regex implementation verified; integration continuing.** All four automata match their original patterns on 10,493 inputs each, plus 505 actual-GSUB boundary cases per profile. Kimi integrated; Qwen integrated and audited; GLM/Llama await merge backend. `tests/test_hf_boundaries.py`. |
| B4 | **Qwen's NFC normalizer is omitted.** Removing NFC alone changes 733 fuzz cases. | **Bounded font-only NFC implemented and audited.** All 5,493 Gutenberg and 4,000 covered fuzz cases pass IDs and widths; all 1,000 Unicode fuzz cases contain unsupported source characters. [Full-font results](qwen-pathway-repaired.md). 4,163 actual-GSUB tests cover Inter closure characters, compositions and combining runs. Explicit eight-nonstarter bound after NFD; longer sequences show `[NFC limit]`. Semantic closure glyphs do not falsely expand cmap coverage. Missing normalized outlines are reported. [NFC results](browser-domain-nfc.json). |
| B5 | **Original Llama 3 has the same conversion losses as GLM.** | **Baseline complete, repair backend in progress.** 0/5,493 Gutenberg and 14/5,000 fuzz exact passes in the old adapted preset; 1,995 fuzz cases unsupported. Vocabulary-rank reconstruction alone cannot reconstruct 678 valid-UTF8 entries. [Results](llama3-pathway-audit.md), [rank audit](rank-canonical-audit.json). |
| C1 | **ctok v3 was restricted to 6,000, then 39,919 pieces.** The repertoire projection discarded multilingual vocabulary. | **Full vocabulary now fits; verification continuing.** Sparse presentation states and shared edge tables include all 48,332 v3 and 14,819 v4.7/v4.8 entries. All 5,493 Gutenberg cases pass for each of v3, v4.7 and v4.8; all 2,103 covered printable Inter singletons also pass. Visible input coverage still depends on the source font. |
| C2 | **ctok's finite search could silently overflow.** v4.7 `~` × 256 exceeded the 256-step budget once framing was included. | **Failure behavior fixed and useful capacity expanded.** Proven graph cuts create independent components without crossing any possible vocabulary edge. Budget is 256 steps per component; unfinished regions show `[limit]`. Whole-paragraph prose can exceed 256 characters while remaining exact. An arbitrary connected component remains bounded. |
| C3 | **ctok long-paragraph layout can be slow.** Old 1,300-character forced browser layout took 50.8 seconds. | **Improved, still a limitation.** New measured times: 130 characters 70.3 ms (was 412.9 ms); 1,300 characters 5.81 s. One sample each, 24px font, 320px column, pre-wrap. Preserving absorbed whitespace clusters was tried and rejected because it worsened both latency and wrapping. [Measurements](ctok-browser-performance-fixes.json). |
| C4 | **Normalized-empty v3/v4.7 inputs lost framing cells.** A space/CR differs from a truly empty string. | **Nonempty cases repaired.** Normalization aliases preserve appropriate content-frame behavior. A zero-character input supplies no glyph on which a font can operate; see E6. |

## Rendering constraints and representation

| ID | Constraint | Resolution or explicit contract |
| --- | --- | --- |
| E1 | **The renderer can erase original Unicode spelling before GSUB.** `cafe\u0301` and `café`, or `A\u030A` and `Å`, can become identical. Byte-level tokenizers distinguish them. | **Unavoidable with the tested font-only/browser interface.** Disabling `ccmp`, disabling `mark`/`mkmk`, and deleting composed cmap entries did not recover the distinction. A font cannot recover information already discarded. Qwen's own NFC semantics are different: they deliberately normalize and can be implemented within a bound. [Tests and explanation](browser-domain-mitigations.md). |
| E2 | **A font cannot inspect outside its shaping run.** An internal bold span changes tokenization of `antidisestablishmentarianism`. Scripts, bidi, fallback, hard breaks and styles can also divide runs. | **Per-run contract.** Uniform typography can reduce style splits, but cannot guarantee whole-message equivalence. Runtime text preprocessing would change the font-only requirement. Browser wrapping is separately tested; whole-run HarfBuzz agreement does not prove correct line layout. |
| E3 | **Full vocabulary is not full glyph coverage.** Inter lacks many CJK and emoji outlines. Ordinary fallback does not have token advances. | **Legitimate invisible coverage expanded.** 113 whitespace/control/format characters now have explicit semantic input glyphs; 452 tests pass IDs/widths. Unsupported visible characters remain reported, not silently blank. Choose a base font with the needed outlines. NFC closure-only characters stay out of cmap and are separately reported when outlines are missing. |
| E4 | **Complex script shaping and variation selectors are not universally covered.** An engine can consume selectors or itemize emoji before substitutions. | **Unresolved renderer scope.** Do not infer complex-script correctness from Latin/LTR tests, or describe a missing-cmap soft-hyphen/ZWJ example as a proven covered-input BPE bug. Unicode/fallback classifications stay in all denominators. |
| E5 | **Independent character drawing discarded source kerning; ctok v3 quote folding changed outlines.** | **Supported kerning and quote outlines repaired.** Default-instance horizontal Latin/DFLT pair advances are baked into atomic/composite tokens and applied to shared parts through required positioning; pads retain 3em token totals. Tests cover TKSP 0/250/500 and prevent cross-token kerning. Unsupported contextual positioning/mark shaping is reported. Equal token widths still intentionally condense long tokens. |
| E6 | **Fonts cannot draw a framing token for actual empty input.** | **Explicit limitation.** There are no input glyphs to substitute. ctok message overhead remains excluded; counts are not a billing estimate. |
| E7 | **ctok is not confirmed Claude segmentation.** Count agreement permits multiple minimum-cost segmentations. | **Reference qualification.** Report exactness against ctok's inferred count model, never hidden Claude token boundaries. No live Claude testing in this audit. |
| E8 | **Special/chat/multimodal framing is outside ordinary-text audits.** | **Explicit scope.** Audits disable automatic special handling and templates. Ordinary source text and model request framing are different inputs. BPE's explicitly supported added-token mode is not evidence of universal chat-template support. |
| E9 | **The old top-level Inter font duplicates tokens at wraps.** | **Retired.** Legacy specimen/downloads are labeled and point to current compiler/default artifacts. The historical binary was not silently replaced. Current default and rebuilt o200k pass the narrow-column reproduction. |
| E10 | **Unicode property versions affect regex classification and normalization.** | **Versioned diagnostics.** Build reports identify the runtime Unicode version; NFC reports its version and bound. Native and browser runtimes must not be assumed equivalent for newly assigned characters without testing. |

## Earlier repairs retained in regression coverage

- OpenAI's four regex frontends, including long-ſ contraction folding: all
  Gutenberg regex controls and actual-GSUB directed/random boundary tests pass.
- OpenAI leading-space clusters prevent the old soft-wrap duplication.
- Inert GPOS and explicit glyph classes prevent fallback mark positioning from
  collapsing token advances; ctok's standalone combining-mark case is covered.
- ctok ideographic spaces, CRLF, punctuation/case seams and leading apostrophe +
  CAPS are covered by follow-up tests.
- All three ctok versions and o200k passed the original Gutenberg repair audit
  before these latest extensions. [Historical repair results](gutenberg-font-fixes.md).
- DeepSeek's latest repaired build passes 5,493 Gutenberg and 4,000 non-Unicode
  fuzz cases; 999 unsupported Unicode and one normalization difference remain.
  [Release verification](deepseek-release-verification.md) uses an immutable compiler snapshot.

## Test inventory and provenance

Each pathway uses all **5,493 contiguous Gutenberg chunks** and **5,000 seeded
fuzz strings verified as exactly 32 reference tokens**, 1,000 each from prose,
code, whitespace, Unicode and punctuation. Unsupported/incomplete cases remain
in the denominator. Raw strings, reference/adapted/font IDs and advances are
retained in compressed JSONL artifacts.

- [Consolidated results](testing-results.md) and [reproduction](pathway-audit.md).
- [Kimi reference verification](kimi-reference-verification.json): pinned
  official Moonshot ranks and regex independently agree on all 10,493 cases.
- [Llama reference verification](llama3-reference-verification.json): original
  Meta algorithm plus public-mirror ranks agrees on all cases. Meta's gated rank
  bytes were not independently authenticated; this is original Llama 3, not 3.1.
- [Browser constraints](browser-domain-mitigations.md), [browser cases](browser-domain-findings.md),
  [OpenAI adversarial baseline](bpe-domain-audit.md).

The user has removed the fixed CPU cap; independent builds and audits can use
additional workers as useful. Shared compiler edits remain coordinated.