# Whitespace audit — 2026-09-26

## Fixed: whitespace classified as emoji

An emoji source can map ordinary whitespace and control characters. The original
Noto Emoji source maps NUL, CR and SPACE. The emoji catalogue previously admitted
these as emoji parts, causing ordinary space-bearing tokens to enter emoji joining
and unnecessarily multiplying glyphs. A full DeepSeek + Inter + original Noto
build needed 72,794 glyphs, exceeding the 65,535 limit. The last release cleaned the
bundled source, but uploaded/custom emoji fonts still exposed the defect.

`EmojiOutlines` now excludes whitespace and Cc controls centrally. ZWJ and emoji
variation/sequence components remain supported. Regression tests use both the
original Noto source and injected aliases for tab, LF, NBSP, narrow NBSP and
ideographic space. The cleaned default font already avoided this issue and does
not need rebuilding. Browser build caches are invalidated for new compiler builds.

Validation: 21 emoji tests, the Noto delivery regression and 44 UI/worker tests
pass. Source ligature, presentation and token-width tests remain intact.

## Browser findings

[Browser report](browser-domain-whitespace-results.md): 60 width comparisons and
20 wrapping cases. A leading ASCII space can become a separate shaping run under
`pre-wrap` or `break-spaces`: ` Hello world` measures 216px rather than 144px at
24px font size. This remains a host rendering limitation. Tabs use tab stops;
newlines consume layout controls instead of painting their token cells. Those
15 differences are retained separately from the 2 leading-space discrepancies.

Tested repeated/trailing/blank spaces, NBSP, narrow NBSP and ideographic space
match whole-run widths. The original soft-wrap duplication example showed no
repeated words or `conf` fragments in screenshots at 180px and 320px.

## ctok and BPE

BPE: six available fonts, 1,398 unique inputs and 2,984 checks each (17,904 total).
Trinity/Laguna pass all checks; DeepSeek has 59 ID mismatches. GLM/Llama retain
353/354 explicit capacity outcomes, respectively. The older o200k artifact lacks
coverage for 1,454 checks; its remaining 1,530 checks pass. These are preserved
as limitations, not counted as successful token comparisons.

ctok: 785 cases per version (2,355 total). v3: 711 pass, 29 count/width failures,
45 semantic-only failures; v4.7: 711/25/49; v4.8: 761/16/8. See
[ctok findings](ctok-whitespace-findings.md) for repair directions and examples.

Detailed tokenizer comparisons are recorded in `ctok-whitespace-audit.json`
and `whitespace-bpe-audit.json`, with individual cases in their adjacent
`.cases.jsonl.gz` files. These are comparisons to local tokenizer references;
no Claude API or CLI calls are involved.

### Unfixed DeepSeek control-separator classification

Two spaces followed by U+001C (`"  \x1c"`) should tokenize as
`[223, 223, 219]`, but the published font emits `[262, 219]`: the two spaces
incorrectly merge. Two tabs followed by U+001C similarly produce `[998, 219]`
instead of `[200, 200, 219]`. Python's `isspace()` includes U+001C–U+001F,
but the tokenizer regex's `\s` does not. DeepSeek's native layout uses the
broader classification, changing whitespace boundaries. This is a compiler
bug, not a browser run boundary. Per-token widths can still be correct while
IDs and total token counts are wrong.

### Unfixed ctok invisible-boundary and normalization issues

Zero-width space, word joiner, BOM and soft hyphen expose lost internal word
markers. For example, apostrophe + U+200B + `a` produces four cells instead of
three in the tested v3/v4.7 fonts. Some v4.8 cases preserve counts but change
the semantic stream, so count-only tests would miss them.

In v3/v4.7, `" \x01 "` should preserve two normalized spaces but the font
retains only one. Its token count happens to agree, again demonstrating why
stream checks are needed. Ordinary whitespace and long runs pass; these are
boundary/normalization cases rather than failures on every space.

The empty-string v3/v4.7 frame remains a known font-only limitation: no input
glyph exists on which to emit a frame. It is retained separately in the audit.