vLLM: serving models fast

Ollama made running AI locally easy: one command, one model, one conversation. But there is a limit beyond which it stops being enough — and knowing where that limit sits is a skill. You notice it when several people use the same model, when you need the model inside an application rather than in a chat, when a single GPU has to do more work than one request at a time.

That is where vLLM comes in. vLLM is an open-source inference and serving engine for large language models, born at the UC Berkeley Sky Computing Lab and today maintained by a community of over 2,000 contributors. It is the engine behind many production deployments, local and in the cloud. In this guide I explain what it is, when it is worth using, how to install it, and the core concepts — throughput, quantization, KV cache — you need to size a setup.

What vLLM is

vLLM is a Python library. You give it a model (downloaded from Hugging Face) and it serves it as an HTTP server with an OpenAI-compatible API: the same endpoints, the same request format, the same responses you get from the official API. That detail matters: every tool built for OpenAI (chats, applications, libraries) works with vLLM by simply changing the address.

Two technologies make vLLM fast.

PagedAttention. To generate a response, the model keeps an account of everything it has seen so far: for each token, the keys and values of attention (the so-called KV cache). This account grows with context length and consumes a lot of memory. PagedAttention manages that memory in small blocks, the way an operating system manages virtual memory. Result: no memory reserved and left unused, and a context that can grow without fragmentation.

Continuous batching. When several requests arrive, vLLM processes them together in a batch. And it does not stop when one request finishes: the next one in the queue enters mid-step, in place of the finished one. The GPU never sits idle, even while waiting for the longest response to complete.

The practical effect is the difference between a server that handles requests one at a time and one that manages many at once: with the same GPU, vLLM can multiply the number of concurrent users and the total output several times over.

When to use vLLM, and when Ollama is enough

vLLM is not Ollama's successor, and it is not "better" in the absolute sense. It solves a different problem.

Use case	Tool
One user, one conversation at a time, zero setup	Ollama or LM Studio
Several people use the same model over a network	vLLM
An application that calls the model via API (assistant, documents, internal service)	vLLM
You want structured output (JSON) and tool calling in your apps	vLLM
You want to measure and maximize a GPU's throughput	vLLM
You just want to chat with a model, quickly	Ollama

The practical rule: one user, Ollama; several users or an application, vLLM. Ollama wins on simplicity (one command, a built-in model library, runs on CPU too); vLLM wins on throughput and integration. The two can coexist: Ollama to try a model, vLLM to put it to work.

A hardware note: vLLM expects an NVIDIA GPU with CUDA (and since recent versions it also runs on AMD, Intel and TPU, and supports Apple Silicon and CPU, but with different performance characteristics). For the classic local use case, an NVIDIA GPU with at least 12-16GB of VRAM is a reasonable starting point.

Installation

vLLM installs as a Python package. The recommended method is with uv, the fast Python package manager:

uv pip install vllm

Or with pip, if you already use it:

pip install vllm

If you do not want to touch Python on your machine, there is an official Docker image:

docker run --gpus all -p 8000:8000 vllm/vllm-openai serve Qwen/Qwen2.5-7B-Instruct

One warning: vLLM is a moving target. The release cadence is very fast (a new minor version almost every month), and each release brings new models, new quantizations, new hardware. Install the current version and read the release notes — the feature you need (a specific model, a specific quantization) may have just landed.

The core concepts for sizing your setup

Throughput. Measured in tokens per second, it is the output of the system. There are two numbers, and they tell different stories: the latency of a single user (how fast your own response arrives, token by token) and the throughput (total tokens generated by all concurrent users). vLLM is optimized for the second: it can sacrifice a little of the speed of each individual response in exchange for a large gain in the number of responses served at once. If there is only one user, that trade-off does not pay off — which is why Ollama is enough.

KV cache and VRAM. The memory a server needs has three components: the model's weights, the KV cache (which grows with context length and the number of concurrent users) and intermediate computations. A 7B model in FP16 takes about 14GB for the weights alone. Quantized to 8 bits (INT8 or FP8) it drops to 7-8GB. On a 24GB card you have room for a 14B model at 8 bits, or a 7B model with a very long context and many concurrent users.

Quantization. Quantizing means representing the model with fewer bits per weight: 16 (FP16, full precision), 8 (INT8/FP8), 4 (INT4, GPTQ/AWQ, GGUF). Fewer bits means less memory used and a faster GPU; the price is a small loss in quality, which varies from model to model and from quantization to quantization. vLLM supports FP8, INT8, INT4 and 4-bit formats like NVFP4 and GGUF. Choosing the right quantization is a trade-off to test in practice, not just in theory: start at 8 bits and drop to 4 bits only when you need more context or more concurrent users.

Batch size and concurrency. The --max-num-seqs parameter caps the number of requests processed in parallel. Raise it if you have many users and lots of VRAM; keep it low if you want each individual response to be as fast as possible.

Running your first model

Here is the minimal sequence to go from zero to an API that answers.

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --served-model-name qwen7b \
  --max-model-len 8192

vLLM downloads the model from Hugging Face on first start, loads it onto the GPU and starts a server on port 8000. To test it:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen7b", "messages": [{"role": "user", "content": "Hello, who are you?"}]}'

The response is a JSON in OpenAI format. From here the rest is simple: point Open WebUI or any other client at http://localhost:8000/v1 and use the model as if it were the OpenAI API.

Practical advice

1. Start with Ollama. Try the model with Ollama (or LM Studio) first: you will understand whether the model fits your use case before investing in a server.
2. Move to vLLM when usage is shared. One user on a laptop: vLLM adds complexity without benefit. A network of three or more machines, or an application that calls the model: vLLM pays off immediately.
3. Measure, don't guess. After installing, test single-user latency and throughput with a few concurrent requests. The numbers tell you whether the hardware fits.
4. Don't over-quantize. 8 bits is the safe default; 4 bits only when you need context or concurrency.
5. Follow the releases. The cadence is fast: a version that did not support a model you care about may have gained it a month ago.

In short

vLLM is the step that turns local AI from "my assistant" into "my service": the same model, the same GPU, but a real server — concurrent, measurable, integrable into any application. When is it worth using? When there is more than one user, or when the model works inside an application. When is it worth staying on Ollama? When you are on your own and having a conversation: in that case, simplicity wins.

Article generated with AI