What Is Pretraining in LLMs?

Pretraining is the foundational phase of building a large language model, where the network learns language, facts, and reasoning patterns by predicting the next token across a vast corpus — trillions of tokens of web text, books, code, and licensed data. It is where parametric memory forms: everything a model "knows" about your brand without searching was absorbed here. GPT-3's pretraining, an early public benchmark, ran on roughly 300 billion tokens (Brown et al., 2020); frontier models since train on far more.

What happens during pretraining?

The model repeatedly reads a passage, predicts the next token, and adjusts its weights when it is wrong — billions of times. No human labels the data; the text itself is the supervision, which is why the phase scales to web volumes. Out of this single objective emerge the model's capabilities and its world knowledge, including the statistical imprint of how often and how consistently your brand appeared alongside its category.

Why is pretraining the window for brand memory?

Because it is the only phase that ingests the open web at scale, and it is infrequent and expensive. A brand described consistently across Wikipedia, its own site, review platforms, and press builds a strong imprint; one mentioned rarely or inconsistently barely registers. Crucially, the imprint freezes at the model's knowledge cutoff — the last date in the pretraining data. Anything published afterward is invisible to that model until a successor is trained.

What does this mean for timing?

Pretraining introduces a structural lag between publishing and parametric recall — commonly 6-18 months, spanning crawl, dataset filtering, the next training run, and model release. Work you do on durable web presence today pays off one or two model generations later, and compounds because the imprint persists once formed. Tracking how each model describes you across releases is how teams see the lag resolve.

Example

A company that earned major press and a Wikipedia article in early 2024 found that models pretrained through 2023 could not describe it at all, while a model released in 2025 summarized it accurately and unprompted. The content did not change between those checks — a newer pretraining run simply caught up. The neighboring phases, fine-tuning and knowledge cutoff, are defined in this glossary.