RAG: ask questions to your own documents

RAG — Retrieval Augmented Generation — is the simplest way to make a local model answer questions about your own documents. You upload your files, you ask a question, and the answer comes backed by the content you uploaded.

Why a model alone is not enough

A language model knows a lot, but it does not know your things: your manuals, your internal policies, your product specs. Ask it about something that was not in its training and it answers anyway — often inventing, with the confidence that is exactly the problem.

On top of that, training has a cutoff date: anything that happened after, and anything that is not public (most business documents are not), it simply does not know.

RAG solves exactly this: before answering, the system looks up the relevant passage in your documents and puts it in front of the model. The model answers from what it just read, not from what it remembers.

How it works, in three components

1. Embeddings. An embedding model turns text into vectors: lists of numbers that represent meaning. Phrases with similar meanings end up close together in this vector space. That is what makes "search by meaning" possible instead of search by exact word.

2. The vector database. The vectors are stored in a database that can quickly answer "which vectors are closest to this one?". The search is by similarity, not by literal match: "brake" and "braking system" are two different words, but their vectors are neighbors.

3. The retrieval loop. When you ask a question, the system converts it into a vector, finds the most similar passages in the database, and builds the prompt with that context added to your question. The model answers having seen the passages.

Before any of that, documents are split into small blocks (chunking): a 200-page manual is not queried whole, but in pieces of a few dozen lines, and only the relevant pieces end up in the prompt.

The full path of one question

1. You prepare your documents and upload them (once, then again whenever they change).
2. The system splits them into blocks and turns them into vectors with the embedding model.
3. The vectors go into the vector database, on your hardware.
4. You ask the question: it is converted into a vector and compared with the stored ones.
5. The closest blocks are inserted into the prompt alongside your question.
6. The model answers using that context.

Everything runs locally: documents, vectors, and model. No copy of your files passes through an external service.

A concrete example with local tools

The shortest path is Ollama for the model and Open WebUI for the documents. You upload your PDFs or texts from the documents section, you pick an embedding model (for example nomic-embed-text, downloadable with Ollama), and the files are split and indexed automatically. Then, in a conversation, you activate "use in current chat" on the document and ask your question: the answer shows where it comes from.

The important point: indexing is one-time work. The heavy lifting (splitting, embedding, storing) happens at upload time; the conversation stays fast, and each question only consults the relevant blocks, not the whole archive.

The limits

RAG is not magic. If the document does not contain the answer, the model will not invent one — provided the prompt lets it. And if retrieval misses the right block, the answer is wrong no matter how good the model is: retrieval quality is the ceiling of answer quality.

It works best with well-written, well-structured documents, and the details matter: block size, embedding model, how many blocks to retrieve. Those are the first things to tune when the results do not convince you.

If you want to understand the individual steps, look at what local AI is and how to pick your first model. And if your goal is to make the model answer from your working documents, Open WebUI is the simplest starting point.

In short

RAG connects a local model to your documents: the model reads first, then answers. Three components — embeddings, vector database, retrieval loop — and no data leaving your hardware. For anyone who has already installed a local model, it is the step that turns the chat into a working tool: real questions, answers backed by your own documents.

Article generated with AI