How to Choose Your First Local AI Model

Choosing your first local AI model is simpler than it looks: you are not looking for the best model in the world, but the right one for your use case and the memory you have. Three criteria — what you need to do, how much RAM or VRAM you have, how much context you want — are enough to get an answer in ten minutes.

The three criteria that matter

1. Use case first, spec sheet second

Pick the model by what it has to do, not by a benchmark ranking:

• Chat and day-to-day assistance: a 7–8B model is more than enough. Conversation quality improves little going from 8 to 14B.
• Text in Italian or French: here the family matters more than the size. Qwen and Gemma write visibly better than the anglocentric families; if language is the point, pick the family first.
• Code: 14B and up, and quantize as little as possible (Q5 or Q8) — code is the first thing that degrades when weights are compressed.
• Chained work (multi-step reasoning, long plans): you need mid-to-large models or "reasoning" variants that think before answering.

2. Memory: the exact calculation

A parameter is on average 1 byte in FP16; quantization compresses it. The practical formula for a Q4_K_M model (4 bits per weight, plus the 6-bit "K" part that makes it slightly larger):

Step	8B	14B	32B
Weights in FP16 (params × 2 bytes)	16 GB	28 GB	64 GB
Weights at Q4_K_M (÷ 3.4 roughly)	~4.7 GB	~8.3 GB	~18.5 GB
KV cache + 8K context	~1–2 GB	~2–3 GB	~3–4 GB
Total to fit in RAM/VRAM	~6 GB	~11 GB	~22 GB

Always add a 20–30% margin for the OS and longer requests. The practical translation:

• 8 GB VRAM: an 8B at Q4_K_M with room to spare; a 14B only at Q3 (not recommended for daily use).
• 16 GB VRAM (or unified memory): 14B at Q4–Q5, or a 32B at Q3–Q4 if speed is not your concern.
• 24 GB VRAM: 32B at Q4–Q5 at useful speed. This is the most versatile tier for quality.

The detail that changes the game: MoE (mixture of experts) models have large total weights but activate few per token. A 30B MoE activates 3: it takes ~19 GB at Q4 like a dense 32B, but runs 3–4 times faster. If your machine sits between two tiers, MoE is almost always the right pick — when a big model is not needed shows it with the numbers.

3. Context

Context is how much text the model remembers in a conversation. Every context token adds bytes to the KV cache: 8K of context on an 8B costs 1–2 GB, 32K costs 4–6. For normal chat, 8K is plenty and costs almost nothing. If you analyze long documents, you need more — and before buying more hardware, look at a RAG system, which attaches documents to the model without inflating the context.

Where to start (October 2026)

Model	Q4_K_M size	Useful RAM/VRAM	Why
Llama 3.1 8B	~4.9 GB	8 GB	The 8B reference: clear Meta license, huge ecosystem, good Italian
Qwen3-8B	~5.3 GB	8 GB	Best quality/multilingual balance at 8B; better Italian and French
Qwen3-14B	~9.5 GB	16 GB	The sharpest quality jump: visibly better reasoning and code
Qwen3-30B-A3B (MoE)	~19 GB	24 GB	32B quality at 8B speed: the right pick with 24 GB

Q4_K_M sizes follow from the rule above (params ÷ 3.4); the Hugging Face model cards list exact file sizes. If you want to learn to read model specs without trusting other people's tables, this guide walks through it.

Your first model, in practice

If you have not installed anything yet, the fastest path is Ollama:

curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3:8b

If the command worked, the conversation prompt appears after a few minutes (first run downloads ~5.3 GB: depends on your network). Type "Explain in two lines what quantization is" and check two things: the answer is in English and well-formed, and the last field on the progress line (tok/s) is consistent with your machine — an 8B settles between 15 and 40 on a GPU, 5 to 15 on CPU, under 4 on a Raspberry Pi. The step-by-step from install to first model covers wiring up an interface and the typical failures too.

If it does not work: ollama run taking minutes without downloading = the model does not fit in memory and is going to swap: drop the size or the quant (ollama run qwen3:8b-q3_k_m). Slow but stable answers = you are fine, that is just your hardware.

And the limits?

Plainly put: a local 8B is not a frontier model. On long multi-step reasoning, complex code and highly articulated writing, the difference with large cloud models is visible — and a 30B MoE narrows it without erasing it. The good news: for most practical jobs (summaries, drafts, document triage, daily assistance, maintenance code), the jump from 8 to 32B is less obvious than you would think, and the 30B-A3B at Q4 fits a single 24 GB card.

In short: start from the use case, apply params ÷ 3.4 plus 1–2 GB of context, and pick the first model that fits your memory with margin. For most machines that is Qwen3-8B or Llama 3.1 8B at Q4_K_M; with 16 GB move to Qwen3-14B; with 24 GB, Qwen3-30B-A3B. The model changes with one command — the machine does not, so it is worth thinking about carefully.

Sources

• Qwen3 — announcement and size tables (Qwen3-0.6B → 32B, MoE 30B-A3B and 235B-A22B, Apache 2.0)
• Meta — Llama 3.1
• Ollama — download and install
• Qwen — quick-start guide

Article generated with AI