What Is Inference in LLMs?

Inference is the runtime phase where a trained large language model generates output — the step that happens every time you ask ChatGPT a question. The model takes your prompt (plus any retrieved documents) as input and produces tokens one at a time, each prediction conditioned on everything before it. Where training sets the weights once, inference runs them, and it is the operation an answer engine performs for every query it serves.

What are the costs of inference?

Two dominate, and both scale with tokens. Latency: generation is sequential, so a longer answer over a longer context takes measurably more time — users feel it as the assistant "thinking." Compute cost: providers bill per token, input and output separately, and at the volume of a consumer search product those pennies compound into the core operating expense. A retrieval pipeline that stuffed entire websites into the model for every query would be both too slow and too expensive to run.

How does inference economics shape retrieval?

The cost structure forces selectivity, which is exactly why publishers must optimize for extraction:

• Engines fetch pages, run main-content extraction to discard boilerplate, and pass only the dense, relevant passages to the model.
• Per-source token budgets cap how much of any one page enters the context, so buried facts get dropped before the model ever sees them.
• Timeouts on fetching mean slow-responding pages are abandoned rather than waited on.

Every one of these constraints rewards content that puts substance early and keeps markup lean.

What is changing with reasoning models?

Reasoning and Deep Research modes deliberately spend more inference at answer time — the test-time compute trend — running extra reasoning steps and additional retrieval rounds. That raises cost per query but pulls in more sources and deeper fan-out, expanding the surface where a well-optimized page can be selected.

Example

Ask a standard model and a reasoning model the same vendor-comparison question: the standard model makes one retrieval pass and cites three sources; the reasoning model spends more inference, fans out into several sub-queries, and cites eight. Same web, different inference budget, different citation opportunity — the mechanics are defined across this glossary.