Quantization

A technique that reduces the numerical precision of a model to decrease its size and speed up inference. 4-bit quantization can shrink a model from 40 GB to about 10 GB while retaining most performance. GGUF and GPTQ are standard formats for quantized models.