Blog
Articles, guides, and insights on local artificial intelligence
9 articles found per "hardware"

Qwen3.8-2.4T-A95B: the math of running 2.4T open-weight parameters, 95B active
The first Qwen-Max-class open-weight model (2.4T parameters, 95B active) outgrows any consumer node: the math on weights, multi-node hardware, 36-month amortization, and power costs, and when running it locally makes sense.

Nemotron-Parse: NVIDIA's OCR family for reading documents locally
Nemotron Parse is NVIDIA's vision encoder-decoder family that reads PDFs locally: formatted markdown, LaTeX formulas, coordinates and a semantic class for every element, in reading order. v1.1 (Nov 2025), v1.2 (Feb 2026) and 2.0 (Jun 2026), all under 1 billion parameters and re-released on October 20; 2.0 reaches 0.72 ANLS on Indic scripts. Locally with vLLM v0.20 through v0.26: your documents stay on your hardware.

Ling 3.1 Flash: 560B open-weight with MIT license and 1M context — what changes if you have 24GB of VRAM
inclusionAI's Ling 3.1 Flash: hybrid MoE 560B total / 25B active, native 1M-token context, FP8 checkpoint, MIT license. What it concretely means for 24GB of VRAM: quantizations, parallelism and API costs.

Aleph Alpha's Kolibri: the 78B MoE that runs (almost) locally
Aleph Alpha's Kolibri (78.1B/3.46B, Apache 2.0, FP8) is the European open-weight MoE: reasoning, tool calling and context up to 1M tokens. In FP8 it weighs ~78GB of VRAM: it runs on an RTX 5090 32GB + shared RAM or on unified-memory Macs; for practical adoption the kolibri1 architecture in llama.cpp and Ollama/LM Studio builds are still missing.

Local Image Models: from SDXL to Qwen-Image
From SDXL's 2.6B parameters to Qwen-Image 2.1's 7B: how diffusion models work, how much VRAM you actually need, and which model to pick for your hardware.

GGUF: The Format That Keeps Models on Your Hardware
GGUF is the single-file format that ships models for local inference: a header with a signature and version, key-value metadata, a tensor index, and weights aligned for fast mmap loading. Quantization types with real sizes (8B from 14.96 GiB at F16 to 4.58 GiB at Q4_K_M), the filename convention, and the rule for picking a size against your VRAM or RAM.

Quantization: Get More From the Same Hardware
Quantization reduces the precision of the weights (from FP16 to 4–8 bit) so models fit in less VRAM and generate more tokens per second. The levels, the real file sizes of Qwen2.5 14B, the rule for choosing from Q4_K_M, and the limits (code, reasoning, languages other than English).

When a Big Model Isn't Needed: Choosing by Size
The method for choosing model size by hardware: the three limits (memory, speed, perceived latency), the VRAM table for Q4 quantization, the hardware classes, and the cases where the small model wins.

Llama 3 and Llama 4: Meta's open family — which model runs on your hardware
Llama 3 and 4 in a table: 3B/8B/70B and Scout MoE 109B, RAM required per model, the Community license and the choice by budget.

