What Is Quantization in Large Language Models?

Quantization is the process of reducing the numeric precision of a language model's weights — for example from 16-bit floating point down to 8-bit or 4-bit integers — so the model needs less memory and runs faster. It is a compression technique applied after training, and it is standard practice everywhere inference cost matters, from cloud APIs to laptops running local models.

How quantization works, briefly

A model's weights are billions of numbers. Storing each at FP16 precision costs two bytes; INT4 quantization cuts that to half a byte, shrinking memory footprint roughly 4x. Formats like GGUF, GPTQ, and AWQ implement this for open-weight models, which is how a 70B-parameter model becomes runnable on a single consumer GPU. The trade-off is rounding error: each weight becomes a slightly coarser approximation of its trained value.

Does quantization change what models know?

Mostly no, occasionally yes. Benchmarks consistently show 8-bit quantization is near-lossless, while 4-bit and below introduce measurable degradation on knowledge-heavy tasks. The damage is not uniform — high-frequency knowledge (famous companies, common facts) is robust, while long-tail facts stored in delicate weight patterns are the first to blur. For a brand mentioned in a handful of sources, an aggressively quantized model may hedge or confabulate where the full-precision version answered correctly.

Why this belongs in a GEO glossary

The AI answers ecosystem is not one model — it is a spectrum of full-precision flagships, quantized API tiers, and local open-weight deployments. Three practical consequences follow. First, your brand's representation varies across deployments of the same model. Second, the growing local-LLM audience (Ollama, llama.cpp users) mostly runs 4-bit quantizations, the least reliable tier for niche recall. Third, measurement done only against a flagship API overstates how consistently models describe you — which is why multi-engine tracking of AI answers beats single-model spot checks.

Quantization is best understood alongside model distillation: distillation trains a new smaller model and can restructure knowledge, while quantization compresses the existing one and mostly preserves it.