OpenAI Current tokenizer sourceHigh confidenceChecked 2026-07-19
The initial result uses o200k_base against the exact text shown in the editor.
Local tokenizer comparison
Compare the same text across AI tokenizers
Paste English, Chinese, multilingual text, or code once and compare it across seven pinned tokenizers.
One inputEnglish, Chinese, and mixed text belong in the same editor.
Plain text onlyNo roles, tools, media, or chat wrappers are added.
No quality rankingFewer tokens do not mean a model is better or cheaper.
Reusable datasetUse the same results in your own analysis
Download the checked benchmark as JSON or CSV. Each row identifies the sample, tokenizer basis, count, and comparison to OpenAI Current.
Editable sample
Text tokenizer comparison
Edit the bilingual starter text or replace it with your own content. Every row receives exactly the same input.
- Characters
- 128
- Lines
- 2
Initial benchmark result
Tokenizer files load only after you run a comparison. Your text stays in this browser.
| Tokenizer | Tokens | Tokens / 100 characters | Difference from OpenAI Current |
|---|---|---|---|
| OpenAI Currento200k_base | 32 | 25 | Same as baseline |
| OpenAI Legacycl100k_base | 39 | 30.5 | +7 (+21.9%) |
| Gemma 3Pinned SentencePiece model | 34 | 26.6 | +2 (+6.3%) |
| DeepSeek V4 ProPinned tokenizer JSON | 32 | 25 | Same as baseline |
| Qwen 3.5Pinned tokenizer JSON | 33 | 25.8 | +1 (+3.1%) |
| Mistral Medium 3.5Pinned tokenizer JSON | 41 | 32 | +9 (+28.1%) |
| Llama 3Pinned tokenizer JSON | 36 | 28.1 | +4 (+12.5%) |
Structured text
Working with JSON?
Whitespace and formatting can change JSON token counts. Use the dedicated JSON token counter to format or minify a payload before comparing it.
For token boundaries and IDs from one tokenizer, use the simpler AI token counter.
How to cite this data
A reproducible plain-text comparison
Every tokenizer receives the exact text shown in the editor. Counts are generated from pinned tokenizer assets or local OpenAI encodings, with no chat roles, system prompts, tools, images, or provider wrappers added.
The dataset is useful for studying tokenization behavior, not for ranking model quality or predicting a complete API bill. Provider APIs may apply additional message formatting or modality-specific accounting.
For a report or project, link to the comparison page and cite the checked date 2026-07-19. The machine-readable files are JSON and CSV.
Next step
Turn token counts into an API cost estimate
Use the measured size of a representative prompt as the input-token assumption in a monthly workload estimate. Add output tokens and request volume to see how the same workload changes across source-tracked models.
Open the LLM API cost calculator or read the cost formula and examples
View tokenizer sources
How StackLens verifies data and calculates cost
OpenAI Legacy tokenizer sourceHigh confidenceChecked 2026-07-19
The initial result uses cl100k_base against the exact text shown in the editor.
Gemma 3 tokenizer sourceHigh confidenceChecked 2026-07-19
The initial result uses Pinned SentencePiece model against the exact text shown in the editor.
DeepSeek V4 Pro tokenizer sourceHigh confidenceChecked 2026-07-19
The initial result uses Pinned tokenizer JSON against the exact text shown in the editor.
Qwen 3.5 tokenizer sourceHigh confidenceChecked 2026-07-19
The initial result uses Pinned tokenizer JSON against the exact text shown in the editor.
Mistral Medium 3.5 tokenizer sourceHigh confidenceChecked 2026-07-19
The initial result uses Pinned tokenizer JSON against the exact text shown in the editor.
Llama 3 tokenizer sourceMedium confidenceChecked 2026-07-19
The initial result uses Pinned tokenizer JSON against the exact text shown in the editor.
FAQ
Tokenizer comparison questions
Can I compare English and Chinese in the same input?
Yes. The editor accepts English, Chinese, multilingual text, code, and punctuation in one input. Every tokenizer receives the exact same string.
Which tokenizer uses the fewest tokens?
It depends on the text. A tokenizer that produces fewer tokens for one input may not do so for another, and token count does not measure model quality.
Are chat templates and system messages included?
No. This comparison encodes only the text in the editor. Roles, chat templates, tools, images, and API request wrappers are excluded.
Why is Claude not in the comparison table?
Anthropic reports an input-token total for a complete API message rather than the same local plain-text measurement, so including it would be misleading.