# ctok whitespace audit

785 cases per version; 2355 comparisons against vendored ctok, using the existing Mutant fonts. No fonts were rebuilt and no Claude calls were made.

| Version | Pass | Count/width mismatch | Semantic-only mismatch |
|---|---:|---:|---:|
| 3 | 711 | 29 | 45 |
| 4.7 | 711 | 25 | 49 |
| 4.8 | 761 | 16 | 8 |

The audit checks counts, total advance, every token cell, normalized semantic output, and missing/unrendered glyphs. Every failure remains in the JSON summary and full gzip JSONL. Font SHA256 hashes are included.

## Findings

- Conventional whitespace and control cases pass: ASCII spaces, tabs, LF, CRLF, NBSP, Unicode Zs/Zl/Zp folding, NUL, stripped controls, and ideographic space. Leading/trailing/interior runs of 1/2/3/8 and long runs through 257 were tested. U+3000 remains distinct rather than folding to ASCII space.
- Default-ignorable format characters break word boundaries: U+200B, U+2060, U+FEFF, and U+00AD are skipped during contextual matching. `hello\u200bworld` produces 4 rather than 3 cells in v3, and 5 rather than 4 in v4.7/v4.8. `a\u200bb` has the right count but loses its internal EOW/BOW markers. This is a font normalization bug within a single HarfBuzz run, separate from browser run partitioning.
- v3/v4.7 head-quote framing also crosses those format characters: `'\u200ba` becomes 4 rather than 3 cells. v4.8 does not have that leading-frame rule.
- v3/v4.7 absorb a raw leading space before normalization exposes a longer space run. For ` \x01 `, the oracle retains two spaces but the font retains one. Folded spaces produce the same sequencing problem (NUL/NBSP/thin-space/line-separator after a raw leading space). These sampled cases retain the same count/width but have incorrect normalized streams.
- Truly empty input yields zero glyphs; v3/v4.7 reference framing requires one cell. This is the existing standalone-font empty-input constraint. Nonempty inputs stripped to empty pass.

The semantic reference includes `ctok.engine.frame_tail`, because long trailing newline runs are priced outside `normalize.stream`. Treating those extra cells as a semantic mismatch would be a test error; the final audit accounts for them and all long-run cases pass.

## Suggested repair direction (not implemented)

Extend the explicit variation-selector word-boundary treatment to other default-ignorable non-word characters, including a guard against quote rules looking across them. Track whether the original first character was ASCII space, then decide whether to absorb it only after stripping/folding reveals the normalized following run. Retain that raw-input distinction: a space exposed by stripping a preceding control must not be absorbed.

Browser collapsing, tab stops, hard line breaks and script/emoji run partitioning are outside this whole-buffer audit and require separate browser tests.

Run: `/tmp/token-mono-gutenberg-venv/bin/python tests/audit_ctok_whitespace.py --workers 2`.