# Token color fonts

The compiler's **Distinguish tokens with** selector offers equal-width spacing,
alternating colors, and consistent token colors. Color modes retain natural
character advances rather than fixed-width token cells. Both are standalone
TrueType color fonts: OpenType substitutions perform the tokenization and
assign COLR/CPAL glyph colors. No text preprocessing is required by the demo.

- **Alternating:** two colors alternate across tokens within a shaping run.
  New runs restart the sequence; a browser can split runs at hard breaks,
  script changes, or other layout boundaries. Invisible whitespace tokens
  participate in the sequence, so visible neighbors can share a color.
- **Consistent:** a token's decoded spelling selects one of the selected 2–16 colors using
  SHA-256. The same token receives the same color independently of position.
  Different tokens can share a color. Leading spaces are part of token identity.

`../token-colors.html` includes ready-made DeepSeek V4.1 Flash × Inter fonts in
both modes. Its editable preview uses the downloaded fonts directly.

## Current scope

Color mode supports the existing BPE compiler backend. ctok and compiled joined
emoji currently fail with an explicit unsupported-mode error. Leave **Use
fallback fonts** unchecked for the supplied Inter examples. Other existing
normalization, source glyph coverage, split-byte, complex-script, and browser
line-layout limitations still apply. Colors do not make tokenization exact for
all tokenizers or all text.

Within-token horizontal source kerning is retained where supported by the
compiler; kerning across token boundaries and the source font's complete
shaping behavior are not reproduced. Color fonts have no token-spacing axis.
Colors are embedded palettes; ordinary CSS `color` does not recolor them.

## Verification

`tests/test_token_color.py` tests both color algorithms, shaping boundaries,
kerning aliases, the glyph limit, and separator cleanup. `tests/test_color_compiler.py`
tests compilation, natural advances, unsupported modes, and long tokens.
`tests/verify_color_fonts.py` compares the shipped fonts with the exact DeepSeek
tokenizer on 250 Gutenberg excerpts, 500 seeded fuzz strings, and nine directed
cases per mode. It checks token pieces, palette assignments, and advances;
`color-validation.json` records the results and font hashes. Browser screenshots
verify actual color rendering; these checks are not a claim of universal
browser wrapping or complex-script support.

## Configurable consistent palette

Select a color count (default 16) and solid background hex (default #1d1a3a).
The compiler filters quantized 8-bit sRGB candidates for WCAG 4.5:1 text contrast,
then greedily maximizes minimum OKLab distance between selected colors. This is
an approximate separation heuristic, not a global optimum or a guarantee that
all users can distinguish all colors. On some backgrounds the readable color
range is narrow. More colors increase font size and can exceed the glyph limit;
the compiler reports an error rather than silently reducing the count.

The build report records the background, actual palette and minimum contrast.
The page previews the compiled font on that background. Other applications must
use a matching background; fonts cannot query the pixels behind their text.
CPAL light/dark palette metadata and CSS font-palette allow application-selected
palettes, not automatic measurement of background color. Contrast is only one
WCAG criterion; color-only token boundaries are not a complete accessibility solution.

Alternating mode has two custom color pickers. Those colors are embedded exactly
as selected, and are not automatically contrast-adjusted.