When a Big Model Isn't Needed: Choosing by Size

"Bigger is better." It's the first thing people think about models, and it holds up to a point. It's also the most expensive idea to follow: a 70B model on consumer GPU hardware can answer more slowly than an 8B model, while a well-chosen 8B covers 90% of the tasks you'll actually run. The right choice doesn't start with "what's the best model overall," it starts with "what's the biggest model my hardware can run well."

This article gives you the method: the three limits that matter (memory, speed, perceived latency), a table for matching size to hardware, and the cases where small models genuinely win.

The three limits that matter

A model's size — its billions of parameters, the "B" — isn't a quality. It's a demand. More B means three things get worse.

1. Memory: how much space it takes

The working rule: params (B) × bits per weight ÷ 8 = GB of weights. At 16 bits (full precision) an 8B takes ~16 GB. With Q4 quantization (roughly 4.8 bits, the most-used local GGUF format) the same model drops to ~5.7 GB. Two important consequences:

• quantization isn't a side compromise — it's the lever that lowers a model's entry bar;
• on top of weights you must add the KV cache (it grows with context) and runtime overhead: keep a 2–3 GB margin.

2. Speed: tokens per second

Local generation is almost always bound by memory bandwidth, not compute. Every generated token requires re-reading the active weights from memory: more active weights, fewer tokens per second. An 8B Q4-quantized model on a 288 GB/s GPU can produce 30–40 tokens/s; a Q4 70B, with about 12× the weights, drops to 2–4 tokens/s on the same card. Quality per token goes up, but time per sentence doubles or triples.

3. Perceived latency

An assistant that takes 3 seconds to answer a short email feels lazy, even if it answers better. The working rule: below ~15 tokens/s, generation starts to annoy in interactive use. This is the limit that retires big models on small hardware before quality ever does.

The practical VRAM rule

For Q4 quantization (the recommended entry point):

Q4 weights	Model size
~3 GB	6–7B
~5.7 GB	8B
~9 GB	14–15B
~19 GB	30B
~22 GB	32B
~44 GB	70B

Add 2–3 GB for KV cache and runtime. This table holds for "dense" models, where every parameter participates in every token. For MoE (Mixture-of-Experts) models, the memory math uses the total and the speed math uses the active count: a 30B MoE with 3.3B active occupies like a 30B but responds like a 3.3B — that's why they exist.

Size class per hardware

Here's the concrete mapping, with models from our catalog and real numbers.

8–12 GB (entry GPUs or laptops with 16 GB shared RAM). Dense 3–8B models at Q4. An 8B Q4 (~5.7 GB) fits comfortably in 12 GB VRAM with generous context; under 8 GB VRAM you drop to 3–4B, still perfectly useful for summaries, rewrites, and first drafts.

16 GB (RTX 4060 Ti 16GB, 4070 Ti Super). This is where the real range opens: 14–18B Q4 (9–12 GB of weights) at mid-to-high context, or small MoE like Qwen3 30B-A3B — ~19 GB of weights but only 3.3B active per token: big-model quality, small-model speed. On 16 GB you run it with moderate context (the KV cache eats the margin).

24 GB (RTX 3090, 4090). The most-used class for local AI. 27–32B Q4 (15–22 GB) at Q4 with 32K context, or larger MoE. It's the first class where a "serious" model for real work — coding, document analysis, long-form drafting — answers at acceptable speed.

32 GB and beyond, or Mac unified memory. 70B Q4 (44 GB: you need 48–64 GB, so this class means 2 GPUs or unified memory), or big MoE. Macs with unified memory change the rules: a 32 GB unified Mac runs models that don't fit on a 24 GB GPU, with speed that depends on bandwidth (410–546 GB/s on recent chips).

When the small model wins

The point of this article: bigger isn't better for you. The small model wins when:

1. The task is short and repetitive. Summarizing an email, renaming files, extracting dates, translating sentences. A 3–8B does these well enough, at 30 tokens/s, with a tenth of the RAM.
2. Latency is part of the experience. Chat, always-on assistant, voice commands. At 3 tokens/s the user stops using the assistant, whatever the answer quality.
3. The context is long but the model is MoE. With a small MoE (3B active) you can afford 32K contexts without the KV cache exploding — something a dense 7B can't sustain.
4. You want to experiment. Running a 4 GB model locally is a 2-minute test; running a 44 GB one is an evening of configuration.

The big model wins when: the task is complex reasoning (math, multi-file coding, legal analysis), latency doesn't matter (batch processing, overnight reports), or the context is huge and dense.

The method: start small, measure, climb

1. Start from the hardware class, not the model name. Find the class that fits your GBs with margin.
2. Pick the smallest of that class for your task. Try the Q4 of the right-class model.
3. MEASURE, don't guess. Run 10 real tasks (your emails, your documents, your usual prompts). If the small one fails where it counts, climb a class. If it never fails, you're done — don't spend more on hardware.
4. Use quantization as the lever, not the excuse. If your class's model doesn't fit, drop a quantization step (Q6→Q4) before dropping a size step.

The exact numbers for every pairing are in the setup planner: select your GPU and model, and it tells you weights, KV cache, and expected speed using the formulas we just explained.

In summary

Model size is a price, not a score. The price is memory, speed, and perceived latency. Pick the class your hardware supports with margin, start at the smallest, measure on your tasks, and climb only when the measurement says so. In almost every case, the model you need is smaller than the one you'd have picked without this method — and it answers faster, with a fraction of the RAM.

Article generated with AI