ttok 1.0 changes the default tokenizer used for text budgets

Simon Willison’s ttok counts tokens and truncates text to a token budget. Version 1.0 changes its default from the GPT-4 tokenizer to the newer GPT-5-family tokenizer. That matters for scripts preparing model inputs. GPT-6 compatibility is provisional: the post cites matching counts across 31 test fixtures, while explicitly noting that OpenAI had not confirmed the tokenizer equivalence.

Simon Willison · Oct 9, 2026

Why it matters

Existing token-budget scripts can produce different results after an otherwise routine tool upgrade.

More context

Try the update against representative inputs and keep model-specific assumptions explicit. The cited experiment is evidence about that corpus, not a guarantee for every possible string.

Keep in mind

A matching token count on sample fixtures does not establish identical tokenization in all cases.

Original read