Before a model can process any input, the text gets broken into tokens, the atomic units it actually reads and generates, usually a whole common word, a fragment of a longer or rarer word, or a punctuation mark. A model doesn't see "unbelievable" as one unit necessarily; it might see it as "un", "believe", and "able" depending on how that model's tokenizer was built.

Tokens are also the unit nearly every AI provider uses for pricing and for context-window limits, so understanding roughly how text translates into tokens (about 1.3 to 1.5 tokens per English word, as a rough rule of thumb) matters directly for estimating cost and for knowing how much content will actually fit into a single request.

Frequently Asked Questions

Is a token the same thing as a word?

Not exactly, a token is often a whole word for common words, but longer, rarer, or compound words frequently get split into multiple tokens, and punctuation and spaces also typically count as their own tokens.

Why do AI providers charge based on tokens instead of words or characters?

Because tokens are the actual unit of computational work the model performs, a token-based price reflects real processing cost more directly than a word or character count would.

How many tokens does a typical business document use?

As a rough estimate, about 1.3 to 1.5 tokens per English word, so a 1,000-word document is typically somewhere around 1,300 to 1,500 tokens, though the exact figure varies by the specific tokenizer a model uses.

Does tokenization work the same way across every language?

No, languages with different structures (long compound words in German, character-based writing in Chinese or Japanese) tokenize differently, which means the same content can use meaningfully more or fewer tokens depending on the language it's written in.

What happens if a request exceeds a model's token limit?

The request either gets rejected outright or the oldest content gets truncated to make room, depending on the system, which is why long documents or long conversation histories sometimes need to be summarized or chunked to fit.

Does an unusual company name or acronym always tokenize awkwardly?

Often, yes, an unfamiliar term can get split into several smaller, less meaningful token pieces, though modern models are generally still able to infer the intended meaning from surrounding context even when a term tokenizes into unusual fragments.