Academy

What quantization actually does: precision loss and vector-space collapse

Quantization means holding a model's weights with fewer bits. But what exactly does this loss of precision do to the model, and why are models so robust to it?

The Felesh team 5 min

Quantization is a mechanism for compressing a model by storing each weight in fewer bits. It reduces memory requirements and accelerates inference. But what happens beneath this saving? What exactly is lost, and why do models typically survive the compression?

Rounding to a smaller grid

At the heart of quantization sits a simple operation: rounding. At full precision, each weight can take one of tens of thousands of possible values. Quantization restricts these to a smaller grid — 256 values in the eight-bit case, or just 16 in the four-bit case. Each original weight is rounded to the nearest available grid point. A weight like 0.3847 might become 0.3861 — a small error, but a real one.

One weight rounded to the nearest grid point, zoomed inA full-precision weight of 0.3847 sits between two points of a 4-bit grid, 0.3805 and 0.3861. The window is magnified 1000 times, so those grid points are 0.0056 apart. Rounding moves the weight to the nearer point, 0.3861, an error of 0.0014. A 4-bit grid has 16 values and an 8-bit grid has 256, a comb 16 times finer.full precision: any value0.38474-bit grid: 16 values, zoomed ×10000.38050.3861error 0.0014: small, but real8-bit: 256 values, comb 16× finer
Rounding snaps each weight to the nearest grid point: 0.3847 becomes 0.3861, an error of 0.0014 that is small but real.

Error accumulation

This small error seems harmless in isolation. But a model has dozens of layers, and each layer’s error cascades into the next. This accumulation magnifies the initial rounding error as it propagates forward. Because of this compounding effect, a coarser quantization grid (fewer bits) degrades quality non-linearly; the loss accelerates.

A rounding error carried through four layersFour of a model’s dozens of layers, left to right. A small rounding error enters layer 1 and is carried into layer 2, which adds its own. The error bar above each layer grows from one layer to the next, and each step is larger than the one before, so the error compounds with depth. The bar heights are illustrative, not measured.error carried forward, layer by layererror compounds with depthlayer 1layer 2layer 3layer 4four of dozens of layers · bar heights illustrative
Each layer inherits the error of the layer before it, so a small rounding error compounds with depth.

The outlier problem

Quantization’s hardest challenge lies elsewhere: outliers. A model’s weight distribution is usually tightly clustered near zero. However, a few very large weights always exist. If you widen the quantization grid to accommodate these large outliers, the gap between grid points grows. Consequently, you lose precision near zero — exactly where the vast majority of weights live. Crude quantization does its worst damage here, which is why advanced methods handle outliers separately.

One outlier widens the grid for every weightTwo panels show the same bulk of weights, clustered between minus 0.4 and plus 0.4. On the left, with no outlier, the grid steps are 0.05 and many grid points fall in the bulk. On the right, one outlier at 2.4 widens the grid: the step grows to 0.32, and most weights now round to 0 or 0.32.bulk onlywith one outliergrid step 0.05grid step 0.32grid step 0.05outlier 2.4most weights now round to 0 or 0.32
One outlier forces the grid wide, so the step between values near zero grows from 0.05 to 0.32.

Vector-space collapse

Under extreme compression (say, two or three bits), a more severe failure mode emerges: distinct values get mapped to a single quantized value. When weights that were previously different become identical, the model loses its ability to discriminate. Its information capacity effectively collapses. This mechanism explains why quality loss at very low bit widths is sudden and severe, rather than a gradual degradation.

Three weights collapse into one grid valueFour distinct weights: 0.31, 0.36, 0.38 and 0.71. A 2-bit grid has four values: 0.00, 0.33, 0.67 and 1.00. Weights 1 to 3 all round to 0.33, and weight 4 rounds to 0.67. Three different weights now share one value, and the model cannot tell them apart.four distinct weights2-bit grid: 4 valuesweight 1: 0.31weight 2: 0.36weight 3: 0.38weight 4: 0.710.000.330.671.00weights 1–3 all become 0.33: the model cannot tell them apart
At two bits, three different weights round to the same value, so the model can no longer tell them apart.

Why models are robust

Despite these mechanics, language models are remarkably robust to moderate quantization. Published measurements show a consistent pattern: at eight bits, quality loss is usually under one percent; at four bits, using a good method, it commonly sits in the one-to-three-percent range. (These are figures reported across the community and model families — not a guaranteed constant; a given model on a given task can sit outside the range in either direction.) Models survive this because their weights are the product of a statistical process, not a rigid computation. Internal redundancy across millions of parameters allows the network to route around small errors. A model typically has more parameters than a task strictly requires, and this margin is what makes quantization viable.

Which precision to reach for

Understanding these mechanics points to a practical default rather than a single correct answer. The available options form a ladder, representing different points on the memory-versus-fidelity trade-off:

  • FP16 (full) — the reference. No quantization loss, maximum memory footprint, and highest cost. This is the baseline everything else is measured against.
  • INT8 — a sensible default: near-baseline quality requiring roughly half the memory. When fidelity matters and the budget permits, this is the safe choice.
  • INT4 (GPTQ/AWQ) — the cost-sensitive rung. It fits significantly larger models onto the same hardware, but only with a good method and only after verification on your specific task. This is where outlier handling and vector-space collapse begin to bite, and some models degrade well past the reported average.
  • GGUF — the format to reach for when serving on a CPU or edge hardware, rather than a datacentre GPU.

Precision is one lever; the serving engine is a separate one. An engine with continuous batching (such as vLLM) increases GPU utilisation independently of the bit width. The two decisions compound; they do not substitute for each other. Furthermore, the operational cost of pushing to four bits is real. You trade a margin of accuracy and take on a per-model verification burden in exchange for memory savings. It is worth doing when a model would not otherwise fit, or when infrastructure cost is the absolute dominant factor; it is rarely worth it if a slightly smaller model running at INT8 can perform the same job with cleaner predictability.

Putting it together

Quantization is a pragmatic trade-off: you buy memory and speed at the cost of a minor loss in precision. Success depends on understanding the mechanical limits — exactly how far a system can be compressed before error accumulation and vector-space collapse degrade its outputs. Ultimately, the final measure of success is always the model’s behaviour on your real task, not the underlying bit count.