What Is Top-p (Nucleus) Sampling?

Top-p sampling — also called nucleus sampling — is a decoding strategy where a language model samples its next token only from the smallest set of candidates whose cumulative probability exceeds a threshold p. With p = 0.9, the model ranks tokens by probability, keeps just enough top candidates to cover 90% of the probability mass (the "nucleus"), renormalizes, and rolls the dice within that set. Everything outside the nucleus — the long tail of unlikely tokens — is cut off entirely.

Why do models sample this way?

The technique comes from Holtzman et al.'s 2019 paper "The Curious Case of Neural Text Degeneration," which documented a dilemma: always picking the single most likely token yields repetitive, looping text, while sampling the full distribution occasionally emits absurd low-probability tokens that derail coherence. Nucleus sampling threads the needle — the tail is dropped, so nothing bizarre appears, but the nucleus is sampled, so output stays varied and natural. Its key advantage over fixed top-k cutoffs is adaptivity: when the model is confident, the nucleus may be two tokens; when many continuations are plausible, it widens automatically.

How does top-p contribute to answer volatility?

Whenever the nucleus contains more than one candidate, identical prompts can take different paths — and in recommendation contexts, the diverging token is often a brand name. If "Asana", "Notion", and "Trello" all sit inside the nucleus after "The best option is ", each run of the prompt is a fresh draw among them. Combined with temperature, retrieval variance, and infrastructure drift, this is why AI visibility must be measured as a frequency across repeated samples rather than asserted from one screenshot.

Example

An API test at top_p = 1.0 (no truncation) produced occasional off-category tool recommendations; at top_p = 0.9 the oddballs vanished but the top-three brand rotation remained. That residual rotation is the honest signal visibility trackers quantify: which brands live inside the nucleus for a prompt, and how often each gets drawn. Temperature, the companion dial, and the measurement practices built on top are defined in this glossary.