Academy

A 70B model on one GPU: a practical guide to quantization

A seventy-billion-parameter model needs about 140 GB of memory at full precision. With quantization you can compress that same model until it fits on a single GPU — and keep quality almost untouched.

The Felesh team 4 min

A seventy-billion-parameter model needs about 140 GB of memory at FP16 precision — an allocation that typically dictates eight 80 GB A100 cards. For most teams, that infrastructure is out of reach. With a technique called quantization, you can compress that same model until it fits on a single GPU. Here, we walk through the memory math step by step to examine what you gain and what you give up.

The memory math

The baseline calculation is straightforward: the number of parameters multiplied by bytes per parameter. At FP16, each parameter requires two bytes, so seventy billion parameters consume roughly 140 GB. Quantization reduces the bits allocated per parameter:

PrecisionBits per parameterMemory for a 70B model
FP1616~140 GB
INT88~70 GB
INT44~35 GB

At INT8, the model footprint halves, fitting cleanly onto a single 80 GB card. At INT4, it drops to roughly 35 GB — small enough for larger consumer hardware, leaving capacity for the KV cache and batch processing.

The weights of a 70B model at three precisionsThree bars hold the weights of a 70-billion-parameter model. At FP16, 16 bits per parameter, the bar is about 140 GB and runs past an 80 GB card. At INT8, 8 bits, the bar is about 70 GB and fits on the card. At INT4, 4 bits, it is about 35 GB, which leaves room on the card for the KV cache.70B weights at each precision80 GB cardFP16 · 16 bits~140 GBINT8 · 8 bits~70 GB · fits the cardINT4 · 4 bits~35 GB · room left
Each halving of the bits per parameter halves the weights: about 140 GB at FP16, 70 GB at INT8, 35 GB at INT4.

What quantization actually does

Quantization rounds each weight to the nearest value on a constrained numerical grid. FP16 provides more than sixty thousand possible values; INT8 offers 256, and INT4 holds just 16. This rounding introduces error. However, language models are fundamentally robust to this degradation. Their scale and internal redundancy allow them to “route around” small errors.

What you give up

The trade-off for quantization is a quality drop that scales with how aggressively you compress:

  • INT8: under one percent quality loss, and usually about twice as fast. In practice, the difference is rarely noticeable.
  • INT4: roughly one to three percent quality loss, provided you use an optimised method. This remains acceptable for most uses.
  • Below INT4 (e.g. INT2): the loss grows sharply; this range isn’t recommended for serious work.

The underlying method determines the severity of this drop. Naive quantization can degrade quality by five to ten percent, while more sophisticated approaches achieve identical compression with minimal penalty.

The outlier problem

Why do crude methods fail? The primary reason is outliers. Among a model’s weights, a few carry very large values. If you stretch the quantization grid to accommodate those extremes, you sacrifice precision across the narrower range where the vast majority of weights reside. Advanced methods are designed specifically to resolve this imbalance.

An outlier stretches the gridA number line from -12.4 to +12.4. Most weights sit near zero, between -0.09 and 0.11. One outlier sits at 12.4. Stretching the 16-level grid to reach 12.4 spaces the levels 1.65 apart, so every ordinary weight rounds to one of two levels, -0.83 or +0.83. The precision left for the bulk of the weights is lost.one outlier sets the grid16 levels, 1.65 apartoutlier 12.4ordinary weights: -0.09 to 0.11-0.83+0.83ordinary weights now share two levels
One outlier stretches the grid, so the ordinary weights, all near zero, collapse onto the same two levels.

The practical recipe

For INT4 quantization on a large model, GPTQ and AWQ are the standard choices. Both manage outliers effectively, keeping quality loss minimal. The NF4 format, used in QLoRA, is similarly optimised for typical weight distributions. The practical sequence is straightforward: try INT8 first. If the model fits your hardware and maintains quality, stop there. If you require tighter compression, move to INT4 using a proven method, and measure the output against your actual tasks rather than public benchmarks.

The practical sequence for choosing a precisionStart by trying INT8, about 70 GB, which fits one 80 GB card. If it fits and quality holds, stop there. If not, move to INT4, about 35 GB, using GPTQ, AWQ or NF4, then measure the output on your own tasks rather than on public benchmarks.try INT870 GB, one cardfits the card?quality holds?yesstopship INT8noINT4 · 35 GBGPTQ, AWQ or NF4measure onyour tasksthe final judge is your own data, not the number on paper
Try INT8 first and stop if it fits and holds quality; otherwise move to INT4 and measure it on your own tasks.

Putting it together

Quantization alters the operational cost equation. Workloads that required eight GPUs previously can now run on one. Success depends on selecting the correct compression threshold — compressing enough to fit the target hardware, not so much that quality collapses. And as always, the final judge isn’t the number on paper but how the model behaves on your real data.