In this article
AI models do not read words. They read tokens, small pieces of text that may be a whole short word, part of a long one, a space or a punctuation mark. Tokens decide three things you care about: how much text fits in one request, how long the answer can be, and what you pay if you use a model through an API. Knowing roughly how many tokens your text holds saves you from cut-off documents and surprise bills.
What a token is
- 1Your text“Explain how photosynthesis works in simple words.”
- 2Split into piecesCommon words stay whole. Rare words break into parts, like photo + syn + thesis.
- 3Each piece gets a numberThe model works on these numbers, not on letters.
- 4Count the piecesThat count is your token count for the prompt.
The exact split depends on the model’s tokenizer. Common English words are usually one token, and rare or long words take several.
The tool estimates tokens from character counts, using about 4 characters per token for OpenAI and Gemini families, 3.8 for Claude and 3.6 for Llama. It is an estimate, not a real tokenizer. For billing-level accuracy, use the token figures your provider reports.
Words, characters and tokens compared
| Text | Characters | Words | Tokens (about) |
|---|---|---|---|
| A sentence like “Explain how photosynthesis works in simple words for a 10 year old.” | 67 | 12 | 17 |
| A short email | 600 | 100 | 150 |
| One page of text | 3,000 | 500 | 750 |
| A 10,000-word report | 57,000 | 10,000 | 14,300 |
| A 300-page book | about 500,000 | about 90,000 | about 125,000 |
At about 4 characters per token. Real counts vary with vocabulary, formatting and language.
Why Hindi and other scripts use more tokens
Tokenizers were built mostly on English text, so common English words are compact and other scripts are chopped into smaller pieces. In the tool’s estimate, every non-English-letter character counts as about 0.9 tokens, against 0.25 for an English letter. A short Hindi phrase of 32 characters is about 29 tokens, while an English sentence twice as long is only 17.
- 67 characters
- About 4 characters per token
- Most words are single tokens
- 32 characters, about half the length
- Almost every character costs close to a token
- Real tokenizers vary, but the gap is real
If you work in Hindi or another Indian language, expect your text to use more of the context window and cost more per idea.
- Plan for more tokens when a prompt or document is not in English.
- Newer tokenizers are getting better at non-English text, so the gap changes with the model.
- Measure with your own text, not an English sample.
Context window: the limit that counts both ways
A model has a context window, the most tokens it can hold at once. It includes your instructions, any pasted document, the conversation so far and the answer it is writing. If you fill the window with input, there is less room for the reply. The tool shows what share of a typical window your text uses, with example sizes of 128,000 tokens for OpenAI and Llama families, 200,000 for Claude and 1,000,000 for Gemini. Actual windows differ by model and version, so check the one you use.
128K tokens is roughly 90,000 English words, or a 300-page book. Hindi text fills it faster.
What it costs
Prices differ widely between models and change often. For a deeper look at running costs across many calls, see the LLM cost guide.
Ways to use fewer tokens
- Cut filler. Remove greetings, repeated instructions and boilerplate from long prompts.
- Trim documents to the relevant part instead of pasting everything.
- Strip extra whitespace and formatting. Tidy text with the word and character limits guide.
- Ask for the length you need. “Answer in 5 bullet points” stops a long reply.
- Reuse a short system prompt instead of repeating a long one each time.
The same habit of checking sizes before you send applies to files: the data storage units guide explains bytes and bits, and the readability guide helps you make a prompt clear. If you do not care about exact counts and just want to know how long a text is, the word counter guide is quicker.
Questions people ask
What is a token in AI?
A token is a small piece of text that a language model reads and writes: a short word, part of a longer word, a space or punctuation. In English, one token is about four characters or three quarters of a word.
How many tokens is 1,000 words?
About 1,300 to 1,500 tokens for typical English text. Technical text, code and non-English languages use more.
AI token kya hota hai?
Token text ka chhota tukda hota hai jo AI model padhta hai, jaise ek chhota shabd ya shabd ka hissa. English me lagbhag 4 characters ya aadhe se zyada shabd (3/4 word) ek token hota hai. Hindi me same baat ke liye zyada token lagte hain.
What is a context window?
It is the maximum number of tokens a model can handle at once, counting your prompt, any documents and its reply. If the input fills it, there is less room for the answer.
Why does Hindi use more tokens than English?
Tokenizers were built mostly from English text, so English words compress well while other scripts are split into many smaller pieces. The exact gap depends on the model.
How do I calculate the cost of a prompt?
Divide the number of tokens by one million and multiply by your model’s price per million tokens. Output tokens are usually priced separately and are often more expensive.
Is this token counter exact?
No. It estimates from character counts. Each model has its own tokenizer, so exact numbers differ. Use your provider’s reported counts for billing.
Is my text sent anywhere when I count tokens?
No. The counting is done in your browser, and the text is not uploaded.