Local AI on Raspberry Pi
A Raspberry Pi is not a server, but it can run local AI models — and for a precise subset of jobs it does the work better than you would think: always-on personal assistant, light automations, classification, prototyping. The point is not speed, it is price: a machine running 24/7 costs less than a coffee a month in electricity. Here are the real numbers and the path from first install to a model that answers.
What you can really do with it, and what you cannot
The physical rule is simple: generation speed is limited by memory bandwidth, not by CPU cycles. The formula:
max tok/s ≈ memory bandwidth (GB/s) ÷ model file size (GB)
The Pi 5 has about 17 GB/s of bandwidth. A Llama 3.1 8B at Q4_K_M weighs ~4.9 GB: the theoretical ceiling is 17 ÷ 4.9 ≈ 3.5 tok/s; measured on real Pi 5 boards it settles between 2 and 3 tok/s (Jeff Geerling recorded 1.99 tok/s on a 16GB). In practice:
| Model (Q4) | File | Pi 5 theoretical ceiling | Pi 5 measured | Use |
|---|---|---|---|---|
| Llama 3.2 1B | ~0.8 GB | ~20 tok/s | 10–15 | automations, classification |
| Llama 3.2 3B | ~1.9 GB | ~8.6 tok/s | 4–6 | light chat, text reorganisation |
| Qwen3-4B | ~2.5 GB | ~6.8 tok/s | 3–5 | chat with better Italian and French |
| Llama 3.1 8B | ~4.9 GB | ~3.5 tok/s | 2–3 | maximum usable quality |
| Qwen3-8B | ~5.3 GB | ~3.2 tok/s | 2–3 | same, multilingual |
3 tok/s means one sentence every 3–4 seconds: fine for interactive chat, not for analysing a 50-page document. MoE models (Qwen3-30B-A3B) do not fit by size: 19 GB at Q4, and the 16GB Pi 5 does not carry it.
Which Pi to buy
Prices changed a lot in 2026: memory cost doubled on AI infrastructure demand, and Raspberry Pi raised prices twice in three months.
| Board | Official price (2026) | Note |
|---|---|---|
| Pi 4 (8GB) | ~$115 | 25 GB/s bandwidth, slower CPU: 1.5–2 tok/s on 8B |
| Pi 5 (8GB) | ~$125 | Works for 1–4B; the 8B Q4 (4.9 GB + KV cache + OS) fits, but barely |
| Pi 5 (16GB) | ~$205 | The right pick: 8B Q4 with margin, 8–16K context |
Official prices are published by Raspberry Pi in its price-rise notes; the values above reflect the two increases (December 2025 and February 2026). Add to the kit: the official 27W power supply (the Pi 5 wants 12–15W under full load), a fan case, and an NVMe SSD if you want decent I/O ($40–60 of accessories).
From first install to a model that answers
1. Prepare the board. Download Raspberry Pi Imager, select Raspberry Pi OS (64-bit), and write the image to the microSD. During configuration, enable SSH and set a name. If you write to NVMe over USB, boot is faster and swap does not wear the microSD.
2. Power on and verify. Connect HDMI, keyboard, network. At first login:
sudo apt update && sudo apt full-upgrade -y
free -h
You should see free RAM (on a 16GB, ~15 GB available). If the Pi reports "undervoltage", you have the wrong power supply: move to the official 27W.
3. Install Ollama.
curl -fsSL https://ollama.com/install.sh | sh
When done, ollama --version should print the version. On a Pi 5 the install is clean: ARM64 is supported directly.
4. Pull the model.
ollama pull qwen3:4b
ollama run qwen3:4b
First run downloads ~2.5 GB: at 50 Mbps that is about 10 minutes. When the prompt appears, try: "Write in English: what is the difference between RAM and VRAM, in two lines."
5. Verify it runs where it should.
ollama ps
It should show the model at 100% CPU. If the model disappears and reappears, it is going to swap: the file does not fit in RAM and speed collapses to 0.5 tok/s. If the model is "100% CPU" and speed is the expected one (3–5 tok/s for a 4B Q4), you are fine.
What it costs in electricity
The Pi 5 under full CPU load consumes 12–15 W; at idle, 3–4 W. The math:
| Scenario | Average draw | kWh/month | Cost (€0.25/kWh) |
|---|---|---|---|
| 24/7, mixed load (avg 8W) | 8 W | 5.8 | ~€1.45/month |
| 24/7, with NVMe and peripherals (avg 15W) | 15 W | 10.8 | ~€2.70/month |
| 12 h/day (avg 8W) | 8 W | 2.9 | ~€0.72/month |
Calculation: 8 W × 24 h × 30 d = 5.76 kWh; × €0.25/kWh = €1.44. That is an order of magnitude below any desktop: a workstation with an RTX 4090 under load draws 350–450 W and costs ~€40/month at 8 hours of work, as shown in What a token costs.
The three tricks that matter
1. NVMe for swap. The Pi 5 supports NVMe over USB: put swap there and the microSD does not wear. The loaded model stays in RAM; the NVMe serves boot and swap of secondary processes.
2. zram to extend RAM. If the model sits at 5.5 GB on an 8GB board, zram compresses free RAM and gives you margin:
sudo modprobe zram num_devices=1
sudo zramctl /dev/zram0 --size 4G
sudo mkswap /dev/zram0 && sudo swapon /dev/zram0
3. Control the context. Every context token adds bytes to the KV cache. On a 4B, 8K of context cost ~1 GB; 32K costs 3–4. For short chats, keep the default (4–8K): you do not need more, and every GB the context takes is a GB less for the model.
Where the Pi is not the right choice
If your case is quality (long reasoning, complex code, articulated writing), the Pi is not the machine: an 8B at 3 tok/s is not an experience. If your case is speed (document analysis, batch automations), 10–20 tok/s on a desktop CPU at €150–200 is 5–8 times faster at a quarter of the price. The Pi wins in one precise case: always on, always available, negligible electricity cost, zero footprint.
In short: the 16GB Pi 5 (~$205) runs Qwen3-4B at 3–5 tok/s and Llama 3.1 8B at 2–3 tok/s, at an electricity cost of €1–3 per month at 24/7. It is an always-on personal assistant, not a workstation. For quality or speed, buy another machine; for "always on", the Pi is unbeatable.
Sources
- Raspberry Pi — official prices and price-rise notes (December 2025, February 2026)
- Tom's Hardware — Pi 5 16GB at $205, second increase in three months
- Ollama — Linux install
- Raspberry Pi Imager — download
- Pi 5 benchmarks: theoretical ceilings and measured values per model
- Jeff Geerling — Llama 3.1 8B on a 16GB Pi 5: 1.99 tok/s
Article generated with AI
In these reading paths
Enjoyed this article?
Subscribe to the newsletter for new tutorials and guides.