GGUF: The Format That Keeps Models on Your Hardware

You downloaded a file with the .gguf extension — probably from Hugging Face — and dropped it into Ollama, LM Studio, or llama.cpp. It works. But what's actually inside that file? Why does a single file of a few GB contain a model that originally took hundreds of gigabytes? This article opens the box: you'll learn what a .gguf file holds, why it's a format built for local inference, how to read a filename, and how to pick the right size for your memory.

If you want the numerical "why" of the compression — what quantization does and where it loses quality — see the article on Quantization. Here we talk about the container, not the math.

What's inside a .gguf file

GGUF stands for "GGML Unified Format." It's the standard file format in which local-inference models are distributed today: a single binary file that bundles everything needed to load and run the model. The name comes from the GGML engine that llama.cpp and most local inference tools are built on.

GGUF is the successor to three earlier formats (unversioned GGML, GGMF, and GGJT). The design is driven by one precise requirement: be unambiguous — contain all the information needed to load the model — and extensible — able to carry new information in the future without breaking existing files.

The file is organized in three blocks, one after the other:

1. The header. It starts with a 4-byte signature holding the string GGUF (the bytes 0x47 0x47 0x55 0x46, read as a little-endian integer give 0x46554747). It's the "tell" that says to the program: "yes, this is a GGUF file." Next come the format version (currently 3, which also introduced big-endian support), the number of tensors in the file, and the number of key-value metadata pairs.
2. The metadata and the tensor index. The metadata are typed key-value pairs: model name, architecture, context length, attention head sizes, tokenizer, and much more. Right after comes the list of tensors: for each one, name, dimensions, data type, and offset (where its raw data starts).
3. The raw tensor data. The actual body of the file: all the weights, already in their final representation (floating-point or quantized), laid out so they can be loaded with a simple memory access.

A detail that matters in practice: the file uses a global alignment (32 bytes by default) and every tensor starts at an offset that is a multiple of that alignment. That's no quirk — it's exactly what the operating system needs to load the file efficiently, as you'll see below.

Why a single file matters

Three properties make GGUF the right format for local use:

• Everything in one file. No separate config file, no standalone tokenizer, no folder of weights. Open the .gguf and you have the complete model, ready. One copy to distribute it; one path to load it.
• mmap-compatible. mmap ("memory map") is the mechanism by which the operating system maps a file into a program's memory without loading it all at once. Thanks to the tensor alignment, llama.cpp can load a multi-GB model simply by "mapping" it: loading is near-instant, and memory is used only for the parts actually needed. That's one reason a quantized model starts in a couple of seconds, even on a slow disk.
• Extensible. New GGUF versions can add metadata or tensor types without invalidating older files. A .gguf you download today stays readable by tomorrow's tools.

Tensors and quantization types

Every tensor has a type, which says how its weights are represented in memory. There are the "classic" floating-point types (F16, BF16) and a family of quantized types: Q8_0, Q4_K_M, Q5_K_M, Q6_K, and so on, up to recent 4-bit formats like MXFP4. The type is a number; the format records it per tensor so the reader knows exactly how to decode it.

How big is each type, really? The table below is llama.cpp's official measurement for a model of roughly 8 billion parameters (Llama 3.1 8B). Rows = type, columns = bits per weight and size in GiB:

Type	Bits/weight	Size (GiB)
Q4_K_M	4.89	4.58
Q5_K_M	5.70	5.33
Q6_K	6.56	6.14
Q8_0	8.50	7.95
F16	16.00	14.96

Read it like this: an 8B model at full precision (F16) takes about 15 GiB; the same model, quantized to Q4_K_M in the same file, takes 4.58 GiB. It's the same network of weights, packed into fewer bits. The exact numbers for the 70B and 405B models are useful when you move up in size: for Llama 3.1, the original file goes from 280.9 GB to 43.1 GB in Q4_K_M, and from 1625.1 GB to 249.1 GB for the 405B model.

How to read a .gguf filename

.gguf files follow a naming convention designed to be read at a glance, made of dash-separated parts. The typical order is: ModelName-Size-Finetune-Version-Quantization[-Type][-Shard]. A real example:

Llama-3.1-8B-Instruct-v2.0-Q4_K_M.gguf

• Llama-3.1 — the base model
• 8B — the parameter class (8 billion)
• Instruct — the fine-tune (a model instructed to follow instructions)
• v2.0 — the version
• Q4_K_M — the encoding, i.e. the quantization level

When a model is too big for one file, it's split into shards with the -00001-of-00005 notation: the first of five, the second of five, and so on. In that case all the files go in the same folder and the tool loads them together.

How to pick a size for your memory

The practical rule that ties this format to your hardware: the file size in GiB, plus a margin for the context cache and the operating system, must fit in the memory you have. For a GPU, "you have" means the VRAM; for a CPU, the RAM.

Start with the model you want and the type you need (Q4_K_M is usually the sensible starting point). Then do two calculations:

• 8B in Q4_K_M ≈ 4.58 GiB of file. With a medium context, it fits comfortably in an 8 GB GPU.
• 70B in Q4_K_M ≈ 43.1 GiB of file. You need a lot more memory, or a CPU+GPU mix that splits the tensors.
• 405B in Q4_K_M ≈ 249.1 GiB of file. Workstation territory, not consumer.

The good news: because the format records every tensor by name and by type, inference tools can spread the tensors across GPU and CPU — the layers that fit in VRAM on the GPU, the rest in RAM — while the file stays single. That's why a model bigger than your VRAM can still run, just a bit slower.

How you get to a .gguf

Two routes, both common. The first is the practical one: you download a ready file from a repository on Hugging Face (many organizations publish the family of quantizations for their model, and the official ggml-org space lets you even build a custom quant). The second, for those who want to do it themselves, is two phases: convert the original model to GGUF at high precision (with convert_hf_to_gguf.py), then apply quantization with llama-quantize, choosing the output type. In both cases the result is that same .gguf file you drop into your tool.

In short

• A .gguf file is everything in one file: weights, metadata, and tokenizer, ready for local inference.
• The header carries the GGUF signature, the version, and the tensor and metadata counts; the tensors are aligned for fast loading via mmap.
• Every tensor has a type (F16, Q8_0, Q4_K_M, …) that determines the file size: for an 8B model, from 14.96 GiB at F16 to 4.58 GiB at Q4_K_M.
• The filename reads at a glance: model, size, fine-tune, version, quantization, and any shard.
• To choose: file size + context cache + system, inside your VRAM or RAM. The format lets you spread tensors across GPU and CPU.

Sources and further reading: the official GGUF specification and the llama.cpp quantization guide, which holds the exact per-type sizes.

Article generated with AI