OpenAI Current tokenizer sourceHigh confidenceChecked 2026-07-19
The initial result uses o200k_base against the exact text shown in the editor.
Local tokenizer comparison
Compare the same text across AI tokenizers
Paste English, Chinese, multilingual text, or code once and compare it across seven pinned tokenizers.
One inputEnglish, Chinese, and mixed text belong in the same editor.
Plain text onlyNo roles, tools, media, or chat wrappers are added.
No quality rankingFewer tokens do not mean a model is better or cheaper.
Editable sample
Text tokenizer comparison
Edit the bilingual starter text or replace it with your own content. Every row receives exactly the same input.
- Characters
- 128
- Lines
- 2
Initial benchmark result
Tokenizer files load only after you run a comparison. Your text stays in this browser.
| Tokenizer | Tokens | Tokens / 100 characters | Difference from OpenAI Current |
|---|---|---|---|
| OpenAI Currento200k_base | 32 | 25 | Same as baseline |
| OpenAI Legacycl100k_base | 39 | 30.5 | +7 (+21.9%) |
| Gemma 3Pinned SentencePiece model | 34 | 26.6 | +2 (+6.3%) |
| DeepSeek V4 ProPinned tokenizer JSON | 32 | 25 | Same as baseline |
| Qwen 3.5Pinned tokenizer JSON | 33 | 25.8 | +1 (+3.1%) |
| Mistral Medium 3.5Pinned tokenizer JSON | 41 | 32 | +9 (+28.1%) |
| Llama 3Pinned tokenizer JSON | 36 | 28.1 | +4 (+12.5%) |
Structured text
Working with JSON?
Whitespace and formatting can change JSON token counts. Use the dedicated JSON token counter to format or minify a payload before comparing it.
For token boundaries and IDs from one tokenizer, use the simpler AI token counter.
View tokenizer sources
How StackLens verifies data and calculates cost
OpenAI Legacy tokenizer sourceHigh confidenceChecked 2026-07-19
The initial result uses cl100k_base against the exact text shown in the editor.
Gemma 3 tokenizer sourceHigh confidenceChecked 2026-07-19
The initial result uses Pinned SentencePiece model against the exact text shown in the editor.
DeepSeek V4 Pro tokenizer sourceHigh confidenceChecked 2026-07-19
The initial result uses Pinned tokenizer JSON against the exact text shown in the editor.
Qwen 3.5 tokenizer sourceHigh confidenceChecked 2026-07-19
The initial result uses Pinned tokenizer JSON against the exact text shown in the editor.
Mistral Medium 3.5 tokenizer sourceHigh confidenceChecked 2026-07-19
The initial result uses Pinned tokenizer JSON against the exact text shown in the editor.
Llama 3 tokenizer sourceMedium confidenceChecked 2026-07-19
The initial result uses Pinned tokenizer JSON against the exact text shown in the editor.
FAQ
Tokenizer comparison questions
Can I compare English and Chinese in the same input?
Yes. The editor accepts English, Chinese, multilingual text, code, and punctuation in one input. Every tokenizer receives the exact same string.
Which tokenizer uses the fewest tokens?
It depends on the text. A tokenizer that produces fewer tokens for one input may not do so for another, and token count does not measure model quality.
Are chat templates and system messages included?
No. This comparison encodes only the text in the editor. Roles, chat templates, tools, images, and API request wrappers are excluded.
Why is Claude not in the comparison table?
Anthropic reports an input-token total for a complete API message rather than the same local plain-text measurement, so including it would be misleading.