# Google preset validation

Both presets use the tested Gemma Unicode BPE backend. These immutable sources
were downloaded again and verified against the locally tested files:

| Preset | Repository | Revision | tokenizer.json SHA-256 |
|---|---|---|---|
| Gemma 4 | google/gemma-4-12B | 023679ed352de9bb66cc873c9009ce3482585c08 | 12bac982b793c44b03d52a250a9f0d0b666813da566b910c24a6da0695fd11e6 |
| Gemini 3.5 Flash local tokenizer mapping | google/gemma-4-E4B-it | ee0ef6023621cff504d758262d4e04895a5af4a2 | cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f |

The source JSON objects differ only in `post_processor`; both ordinary-text
oracles run with `add_special_tokens=False`. This establishes equivalence for
the open local tokenizer mapping, not the behavior of a hosted Gemini model or
its message framing. No cloud model calls were made.

The earlier [full Gutenberg audit](gemma-gutenberg-audit.md) passed all 5,493
chunks in both normal and lite mode. The new `tests/audit_google_presets.py`
replays the cached normal font against both references, adds 500 seeded ASCII
inputs and directed whitespace/Unicode cases, and retains all failures in its
JSON report. These cached full fonts predate recent fallback/emoji changes;
they are not claimed to be fresh builds of the current compiler.

At the earlier preset-label audit, current-source tests: `python -m unittest discover -s tests -p 'test_gemma*.py' -v`
passed all 12 tests, including ordinary compiler behavior and Noto/Mutant emoji
joining with unchanged byte-token cells and spacing.

Remaining scope limits include HarfBuzz normalization of decomposed accents,
base-font coverage, complex-script shaping, and browser run/line boundaries.
A subsequent fresh rebuild fixes the reproduced Chromium duplicated-word bug
in both modes and passes all 5,493 Gutenberg chunks again. Narrow-column overhang
remains; see [wrap validation](google-wrap-validation.md). Tested preset names need not carry
a blanket Experimental suffix, but these specific limitations remain relevant.

## New replay results

Both reference tokenizers agree on **all 6,007 inputs**. The cached normal font
passes all **5,493 Gutenberg chunks**, all **500 seeded ASCII inputs**, and
**10 of 14 directed inputs**, including whitespace runs and literal token names.
The four retained exceptions are:

- Decomposed `cafe\u0301`: HarfBuzz composes the accent, changing token IDs.
- `a\u3000b`: the historical font shapes the ideographic space as the ordinary
  space marker; the reference expects three byte-fallback tokens for U+3000.
  This is a cached-font finding, not a fresh-current-build result.
- `🙂` and `👍🏽`: the historical Inter-only font has no emoji coverage and emits
  `.notdef`. Current compiler fixture tests separately verify emoji fallback.

No width errors or new ordinary ASCII/Gutenberg mismatches occurred. See
[exact results and file hashes](google-presets-validation.json).

The ideographic-space exception is **missing coverage in the historical font**:
its cmap has no U+3000, allowing HarfBuzz's space fallback. A new current-compiler
regression, `tests/test_google_whitespace_coverage.py`, passes: U+3000 is present
in cmap and `a\u3000b` retains both ordinary tokens and all three reference byte
cells, each exactly 3 em. That added one passing current-source test to the 12
compiler/emoji tests in the earlier audit; a full rebuild subsequently passed
the wrapping-fix follow-up below.

## Fresh wrapping-fix follow-up

The cached replay above remains historical evidence. The current full normal and
lite builds pass 5,493/5,493 Gutenberg cases, with 341,367 exact reference tokens
per mode and no token-width failures. The focused current-source suite now has
19 Gemma tests plus the separate ideographic-space test. Normalization and missing
source glyph coverage remain separate from the corrected duplicate-ink defect.