What Is Tokenization in LLMs?

Tokenization is the process of converting raw text into the token sequence a language model actually processes. Before an LLM reads anything — your prompt, your web page, your brand name — a tokenizer segments it using a fixed vocabulary learned from training data. The dominant method is byte-pair encoding (BPE): start from characters, repeatedly merge the most frequent adjacent pairs, and stop at a target vocabulary size — around 200,000 tokens in OpenAI's o200k_base vocabulary used by GPT-4o.

How does tokenization decide where text splits?

Frequency in the training corpus decides everything. Strings the tokenizer saw constantly — "search", "email", "the" — earn single tokens. Rarer strings fragment along whatever sub-pieces exist in the vocabulary. Tokenization is also case- and space-sensitive: " Apple" (leading space), "Apple", and "APPLE" can tokenize differently, and non-English text generally fragments into more tokens per word, which is one reason multilingual processing costs more.

Why do some brand names tokenize badly?

An invented name like "Xyloflux" was never frequent in any corpus, so it shatters into fragments such as "X-ylo-flux." Consequences are subtle but real:

• The model must learn that a multi-token sequence denotes one entity, which takes more evidence than a single-token name requires.
• Unusual casing and punctuation (mid-caps, dots, hyphens) multiply surface forms — each variant tokenizes differently, splitting the statistical evidence across spellings.
• Names that collide with common words (a startup called "Sage" or "Amber") tokenize cleanly but inherit an entity-disambiguation problem instead.

None of this is disqualifying; models handle multi-token entities constantly. It simply raises the value of consistent, high-frequency usage.

What are the naming implications?

Pick one canonical spelling and casing and enforce it everywhere — site, docs, press, review profiles — so every mention reinforces the same token sequence. Avoid gratuitous stylization, and always co-locate the name with category context ("Menra, the AI visibility platform") so the fragments bind to meaning. These are the same moves that build a clean brand entity, covered elsewhere in this glossary.

Example

Run any brand name through the open-source tiktoken library and you can see the split directly: "Salesforce" is a familiar single unit to modern vocabularies, while a newly coined SaaS name may emit four fragments. The four-fragment brand isn't doomed — it just needs the web to say its name the same way, many times.