What Is the Transformer Architecture?

The transformer is the neural network architecture underlying essentially every modern large language model. Introduced by Vaswani et al. in the 2017 paper "Attention Is All You Need," it processes an entire sequence of tokens in parallel, using an attention mechanism to weigh how much each token matters to every other. GPT — Generative Pre-trained Transformer — wears the lineage in its name; Claude, Gemini, and Llama share it.

What problem did the transformer solve?

Earlier language models (RNNs, LSTMs) read text one token at a time, struggling to hold long-range relationships and resisting parallel computation. The transformer discarded recurrence: attention lets any token consult any other directly, no matter the distance, and the whole computation parallelizes across GPUs. That unlocked training on web-scale corpora — and scale, it turned out, was the ingredient. The architecture's efficiency is what made models with hundreds of billions of parameters economically trainable.

How does a transformer process language?

Three moves, simplified:

1. Tokenize and embed — text becomes tokens; each token becomes a vector encoding its meaning.
2. Attend — stacked attention layers let every token's representation absorb context from the rest of the sequence, resolving what "it" refers to and which sense of "apple" applies.
3. Predict — the model outputs a probability distribution over the next token, and generation repeats the loop.

Everything an assistant does — answering, summarizing, recommending your competitor — is that loop running at scale.

Why did the transformer make AI search possible?

Two downstream consequences. First, scaling: transformers trained on trillions of tokens absorbed enough world knowledge to answer questions directly, creating parametric brand knowledge. Second, embeddings: the same architecture family produces the vector representations that power semantic retrieval, letting engines match a user's question to your page by meaning rather than shared keywords. Modern answer engines are transformers at both ends — one embedding and retrieving, one reading and writing — which is why content strategy now optimizes for how transformers parse text, a thread running through this entire glossary.

Example

When Perplexity answers "alternatives to Salesforce for nonprofits," a transformer-based embedding model retrieves candidate passages by semantic similarity, and a transformer-based generator composes the answer from them. One 2017 architecture, applied twice, replaced the ten blue links.