Quantization: Get More From the Same Hardware

You have a 16GB VRAM GPU and a 14-billion-parameter model that, at full precision, needs about 28GB. The model won't fit. And yet: almost always, a quantization is all it takes to make it run, with a quality loss you won't even notice.

Quantization is the technique that makes running large models on consumer hardware possible. In this article you'll understand what it actually does, where it loses quality, and how to pick the right level — with a concrete example on a real model.

What quantization is

A language model stores its parameters as numbers. In training it saves them at high precision: typically 16-bit (FP16) or 32-bit (FP32). Each parameter takes 2 or 4 bytes, and multiplied across billions of parameters the size becomes enormous.

Quantization reduces the precision of those numbers. Instead of 16 bits per parameter it uses, say, 4. The file gets smaller and, as a consequence, takes less memory and runs faster, because the GPU reads fewer bytes from VRAM.

No "smaller model" is trained: it is the same model, the same weights, just represented with fewer bits. It is compression, not a different architecture.

Why VRAM is the real limit

Generation speed depends on memory bandwidth: at every token, the GPU must re-read the weights. Fewer weights, fewer bytes to move, more tokens per second. Quantization therefore wins on two fronts:

1. Memory — the model takes less VRAM, leaving more for the context cache.
2. Speed — on the same GPU, the quantized model generates more tokens per second.

On a memory-limited GPU (12–16GB) it is often the difference between "it runs" and "it does not run at all."

The levels: Q2, Q4, Q5, Q8

Files for local inference are almost always in GGUF format, the de facto standard that came with llama.cpp. The suffix in the filename indicates the quantization level:

• FP16 — full precision. The reference, but the largest.
• Q8_0 — 8-bit. Quality nearly indistinguishable from FP16.
• Q6_K — 6-bit. Still nearly identical, with a smaller file.
• Q5_K_M — 5-bit. Excellent quality/size trade-off.
• Q4_K_M — 4-bit. The de facto standard: most local setups start here.
• Q3_K_M — 3-bit. When VRAM is genuinely scarce.
• Q2_K — 2-bit. Extreme: quality degrades noticeably.

The "K" stands for "K-quant", a more sophisticated quantization family that distributes bits non-uniformly, protecting the most important parameters. That is why a Q4_K_M almost always renders better than a Q4_0 at equal bits.

A concrete example: Qwen2.5 14B

Take Qwen2.5-14B-Instruct, a 14-billion-parameter model widely used locally. The real file sizes of the official GGUFs from the Hugging Face repo:

Level	File size	Perceived quality
FP16	~28 GB	Reference
Q8_0	~15.7 GB	Nearly identical
Q6_K	~12.1 GB	Nearly identical
Q5_K_M	~10.5 GB	Excellent
Q4_K_M	~9.0 GB	Very good (the standard)
Q3_K_M	~7.3 GB	Good, with dips on hard tasks
Q2_K	~5.7 GB	Poor, casual use only

On a 16GB GPU (RTX 4060 Ti 16GB, RTX 5080), Q4_K_M leaves about 7GB free for the context cache — room for tens of thousands of tokens. On a 12GB GPU (RTX 3060 12GB), Q4_K_M barely fits with a short context, while Q5_K_M does not: this is the point where the level stops being an aesthetic choice and becomes a necessity.

The practical rule: start at Q4_K_M, move up to Q5_K_M or Q6_K if the memory is there and you need more quality, drop to Q3_K_M only if there is no alternative.

How to choose in practice

1. Start from the model you need (see how to choose your first model) and check how many GB it takes at Q4_K_M.
2. Compare with your VRAM, keeping 2–4GB in reserve for the context cache.
3. If it does not fit, drop a level. If it fits with headroom, consider moving up.
4. Test. Perceived quality depends on the model and the task: for summarizing documents Q3_K_M is often enough; for reasoning or code, Q4_K_M and up.

With Ollama the level is implicit in the tag: ollama run qwen2.5:14b pulls the Q4_K_M variant, the library default. With llama.cpp or LM Studio you pick the GGUF file explicitly and have full control.

The limits to know

• It is not free. Below Q4_K_M the loss becomes real, especially for: code, multi-step reasoning, generation in languages other than English.
• Context weighs. A model that "fits" at short context may not fit at long context, because the key-value cache grows with the tokens.
• Every model reacts differently. A 7B at Q3_K_M can be more useful than a 14B at Q2_K. Size matters, but quantization modulates it.

In short

Quantization is why a 14B runs on your 16GB. Understanding it gives you control over a trade-off you would otherwise manage by feel: quality versus memory and speed. Start at Q4_K_M, measure against your VRAM, and adjust one level at a time.

Want to see how these numbers translate into choosing a model for your hardware? Continue with how to read a model's specs and when a big model isn't needed.

Article generated with AI