Ollama vs llama.cpp: under the hood

You've probably wondered how a language model actually runs on your machine. The answer almost always goes through a single engine: llama.cpp. And most people who run Ollama don't know that Ollama uses llama.cpp under the hood. But the two tools are not the same thing. One is the engine, the other is the car: a powerful engine to install, versus a car that adds the wheels, the dashboard and the automatic gearbox.

In this article we open the bonnet. We see what's underneath, when the difference actually matters, and how to choose with a clear head — no slogans.

What llama.cpp is

llama.cpp is an inference engine written in C/C++: it runs language models on your hardware — CPU, consumer GPU, Apple Silicon — with minimal configuration and state-of-the-art performance. It is built on the ggml library and its stated goal is LLM (and VLM) inference with a minimal setup and top-tier performance, locally or in the cloud. The llama.cpp repository and README describe the vision and the capabilities.

The strength of llama.cpp is control. It gives you:

• 1.5/2/3/4/5/6/8-bit integer quantization, to cut memory usage and speed up inference.
• Hybrid CPU+GPU inference: you put on the GPU only the layers that fit in VRAM and the rest stays in RAM, so models bigger than your VRAM still run.
• A full toolset: llama-cli for chatting, llama-server (now also llama serve) for a server with an OpenAI-compatible API and a built-in web UI, llama-quantize for converting weights, llama-bench for measuring tokens per second, llama-imatrix for smarter quantizations.
• Thousands of flags: threads, offloaded layers, context, flash attention, speculative decoding, JSON grammars, and more.

In practice, with llama.cpp you touch every screw: each behavior is a flag you decide. It's the right tool for people who want to squeeze the last drop of speed or build custom inference pipelines.

What Ollama is

Ollama is a runtime one level up, written in Go. It's a daemon that runs in the background and layers three things on top: a packaging model, a model library, and an API. The Ollama repository and the official documentation detail the installation and the commands.

• ollama pull qwen3 downloads the model from the Ollama library: one name and you're running, without hunting GGUF files, choosing quantizations, or managing chat templates.
• ollama run qwen3 starts the chat in one shot, with a simple command-line interface.
• The local server exposes a native API and an OpenAI-compatible one, so your tools and apps can connect without knowing a local model is behind them.

The key technical point: for most local inference, Ollama uses a forked, patched version of llama.cpp. So the two projects are not interchangeable builds of the same tool: Ollama adds packaging, downloading, templates, process management and API around the engine. When a new model drops, Ollama supports it once the maintainers update the llama.cpp they run internally.

The difference, in one sentence

llama.cpp is the engine; Ollama is the car that mounts it and gets you moving. Same heart, two different philosophies.

	llama.cpp	Ollama
What it is	C/C++ inference engine	Managed runtime (Go)
Setup	Binary or build from source, then flags by hand	One install, then ollama run
Models	GGUF files you download yourself (Hugging Face)	Library with ollama pull
API	llama-server, OpenAI-compatible	Native API + OpenAI-compatible
Interface	CLI + server web UI	Simple CLI, no official GUI
Control	Total: every flag, custom builds	Sensible defaults, less tuning
For whom	Tinkerers, max performance, custom setups	Fast start, developers, daily use

Does speed matter?

This is where the myth to debunk lives. Since Ollama runs on llama.cpp, the engine is the same: with equal model, quantization and flags, the pure speed difference is small. The real differences come from two things:

1. The defaults. Ollama picks context, GPU layers and other parameters for you. If its defaults are not your ideal, you leave speed on the table. With llama.cpp, you decide.
2. The exact model. Ollama serves its own builds of models; you may get a slightly different quantization or architecture than the GGUF you'd load manually on llama-server.

The practical rule: for daily use (chat, tools, prototyping) Ollama's speed is perfectly fine. If you're benchmarking, or you have a big model you want to push to the limit, or you're optimizing for specific hardware, then llama.cpp's control becomes a measurable advantage — and this is where llama-bench gives you the numbers to know exactly what you're giving up.

How to choose (no slogans)

Start with Ollama if you want to be running in minutes, you're plugging in an existing app or tool via API, and you don't want to think about GGUF files, templates and flags. It's the right path for the majority of cases, and it's the de-facto standard for running local AI in lightweight production.

Drop down to llama.cpp if you want maximum performance on specific hardware, you're using speculative decoding or advanced quantizations, you're building a service with the OpenAI API without the daemon's extra layer, or simply you enjoy opening the bonnet. Moving from Ollama to llama.cpp is low-risk: both expose an OpenAI-compatible endpoint, so your applications' API doesn't change.

Can they coexist? Yes. Many people use Ollama for daily work and llama-server when they need the control. Nothing stops you from running both on the same machine.

In practice: the commands

With Ollama:

ollama pull qwen3
ollama run qwen3

With llama.cpp:

llama-server -m qwen3-q4_k_m.gguf -c 32768 -ngl 99
# or, with direct download from Hugging Face
llama serve -hf ggml-org/qwen3:Q4_0

In the second case you choose the context (-c), the GPU layers (-ngl) and the quantization (the .gguf file), while in the first Ollama handles everything for you.

The big picture

Ollama and llama.cpp are not rivals: they are two layers of the same stack. If you're curious about local AI, Ollama is the simplest entry point; llama.cpp is the exit door when you want real control. Understanding that one contains the other removes the "which one is better" doubt and gives you a more useful question: how much control do I need today, and what does it cost me to have it.

The two don't exclude each other. Start with the simple runtime, and when a limit asks for more, open the bonnet: you'll find everything you need already there.

Sources

• llama.cpp — repository and README
• Ollama — repository
• Ollama — official documentation

Article generated with AI