What Is a Token in an LLM?

A token is the atomic unit of text a large language model reads and writes — a chunk of characters, usually a word, part of a word, or punctuation mark. LLMs never see letters or words directly; a tokenizer converts text into a sequence of token IDs, the model predicts the next token ID, and a decoder turns IDs back into text. In English, one token averages roughly four characters, or about three-quarters of a word.

What does a token look like in practice?

Common words map to single tokens ("the", "search"), while rarer or compound strings split: "optimization" may be one token in one vocabulary and three in another, and an unusual brand name can shatter into several fragments. OpenAI's GPT-4o tokenizer (the o200k_base vocabulary, used via the open-source tiktoken library) contains around 200,000 distinct tokens; every text the model handles is spelled from that fixed inventory.

Why do token limits shape AI answers?

Everything in an LLM pipeline is budgeted in tokens:

• Context windows are token-denominated — a model with a 128,000-token window can hold only so much retrieved content, so pipelines pick passages, not sites.
• API pricing is per token, in and out, which pushes engines toward aggressive extraction: strip boilerplate, keep the dense parts, truncate the rest.
• Fetch processing has practical ceilings — a page whose useful content sits after thousands of tokens of navigation markup risks having key facts fall outside what the pipeline keeps.

The upshot for publishers: front-load the substance, keep markup lean, and make every retained token count.

Example

A 2,000-word product page is roughly 2,700 tokens of prose — but shipped with 90 KB of HTML scaffolding, the raw document runs many times that. An answer engine extracting main content under a token budget will keep the article body and drop the rest; if pricing details live in a footer widget rather than the body, they were spent tokens the pipeline never kept. How text gets split — and why some brand names tokenize badly — is covered under tokenization in this glossary.