Ready
Loading tokenizer...
Tokenized text will appear here as you type.
Your text stays in this browser. Nothing you paste is uploaded.
Tokenizer and API totals may differ when request formatting or model-specific processing adds tokens.
AI Tokenizer & Token Counter
Count text with OpenAI, Gemini, Gemma, DeepSeek, Qwen, Mistral, Llama, or Claude. Inspect token boundaries and IDs when the selected tokenizer provides them.
How counting works
OpenAI, Gemini, Gemma, DeepSeek, Qwen, Mistral, and Llama tokenization run locally on plain text in your browser. Their token boundaries come from the selected tokenizer rather than from splitting text on spaces. Claude counts come from Anthropic's official token-count API through a server-side request.
Claude reports the input-token total for a complete user message and does not return token pieces or token IDs. That total may include message formatting and tokens Anthropic adds for system optimizations, so it is not directly comparable with the plain-text OpenAI or Gemini counts.
- OpenAI Current
- Current OpenAI text encoding
- OpenAI Legacy
- Legacy OpenAI text encoding
- Gemini
- Gemini-compatible local SentencePiece tokenizer
- Gemma
- Official Gemma 3 tokenizer running locally
- DeepSeek
- Official DeepSeek V4 tokenizer running locally
- Qwen
- Official Qwen 3.5 tokenizer running locally
- Mistral
- Official Mistral Medium 3.5 tokenizer running locally
- Llama
- Llama 3 tokenizer running locally
- Claude
- Official Anthropic API message input count for Claude Sonnet 5
For request-volume and output-token calculations, see the monthly LLM API cost guide. Using OpenAI? Check the current OpenAI API token prices.
To compare the same English, Chinese, JSON, and code samples, open the reproducible tokenizer comparison.
Using Gemini? Open the focused Gemini tokenizer.
Want another tokenizer? Send a request.
FAQ
Tokenizer questions
What does an AI tokenizer do?
A tokenizer splits text into the smaller units a language model processes. A token can be a whole word, part of a word, punctuation, whitespace, or another character sequence.
Why can token counts differ between models?
Model families can use different tokenizers. The same text may therefore produce different token boundaries, token IDs, and total counts.
What are token IDs?
Token IDs are the numeric identifiers assigned to token pieces by the selected tokenizer. The IDs are specific to that tokenizer and should not be treated as universal across models.
Does the tokenizer include API message formatting?
No. This page tokenizes the text you paste. An API request may add message formatting, tools, images, or other provider-specific tokens.
Does my text leave the browser?
OpenAI, Gemini, Gemma, DeepSeek, Qwen, Mistral, and Llama modes run locally. Claude mode sends the text through StackLens to Anthropic's official token-count API. StackLens does not store the submitted text.
Is the Gemini mode an official Gemini API count?
No. It provides Gemini-compatible plain-text tokenization. Gemini API usage can differ because of message formatting, tools, media, and model-specific processing.
Can I inspect Claude token IDs?
No. Anthropic's official token-count API returns the input-token count for a complete message, but it does not return individual token boundaries or token IDs. The count may include message formatting and Anthropic-added system tokens, so do not compare it directly with a plain-text tokenizer count.