RAG pipeline: how documents become an answer with a source.
A RAG pipeline makes a language model answer from your documents instead of from memory. This guide explains the RAG architecture block by block: chunking, embeddings, vector database, hybrid search, reranking and RAG evaluation, with starting values, common mistakes and a checklist.
For IT and business teams · as of 5 November 2026
Key points
- A
RAG pipeline has two parts:indexing (read documents in, split into chunks, embed, store) andquerying (search, rerank, generate an answer with a source). - Most quality problems happen before the language model: poorly prepared documents, unsuitable chunking or a search that misses the right passage.
Hybrid search andreranking usually improve hits noticeably, especially for technical terms, numbers and names.- Without
RAG evaluation on a fixed set of test questions, you cannot tell whether a change helps. Measure retrieval and answers separately.
What is a RAG pipeline?
RAG stands for retrieval-augmented generation: before every answer, the system finds the right passages in your documents and passes them to the language model. The
The first part runs in the background: documents are read in, split into sections and stored as vectors. The second part runs with every question: search, choose the best hits, generate an answer with a source.
RAG architecture: eight building blocks at a glance.
Every block has settings that matter. The table shows what counts and which mistake is most common at each step.
| Block | Job | What matters | Most common mistake |
|---|---|---|---|
| 1 Connect sources | Fetch documents from drives, wikis, websites | Carry permissions over, sync changes | Stale copies in the index |
| 2 Prepare | Extract text from PDF, Word and HTML | Handle tables, headings and headers cleanly | Footers and menus in the text |
| 3 Chunking | Split text into searchable sections | Size, overlap, splitting at headings | Sections cut off mid-sentence |
| 4 Embedding | Turn sections into vectors | A multilingual model for German texts | Switching models without reindexing |
| 5 Index | Store vectors and search them fast | Metadata such as source, date, rights | No source stored with the chunk |
| 6 Retrieval | Find passages matching the question | Top-K, threshold, hybrid search | Too few or too many hits |
| 7 Reranking | Score hits more precisely and reorder | Number of candidates and latency | Best hit ends up in ninth place |
| 8 Generation | Write the answer from the passages, with a source | Prompt, context window, allow “I don’t know” | Answer without evidence |
RAG chunking: splitting documents the right way.
Chunking decides what the search can find at all. A chunk should hold one statement that makes sense on its own. Too small and the context is missing; too large and the search gets diluted and the context window fills up.
| Strategy | How it works | Good for |
|---|---|---|
| Fixed size with overlap | Cut every 300 to 800 tokens, overlap 10 to 20% | Getting started, uniform texts |
| By structure | Cut at headings, paragraphs or sections | Manuals, policies, contracts |
| Semantic | Cut where the topic changes | Long, loosely structured texts |
| With context header | Put the document title and heading in front of every chunk | Many similar documents |
Hybrid search and reranking: finding the right passages.
Choosing a vector database for RAG (self-hosted).
The
The criteria are vendor-neutral. Which option fits depends on data volume, team and infrastructure.
| Criterion | What to look for | Question to settle first |
|---|---|---|
| Deployment | Embedded, the database runs as a library inside the application process: little to operate, good for a single server. As a separate server, several applications share one index and scale independently, but there is one more service to run and monitor. | How many applications and servers use the index? |
| Persistence and backup | The index must survive restarts and be backed up. It can always be rebuilt from the sources, but with large collections that costs time and compute for the embeddings. | Are there snapshots, and have you tested a restore? |
| Filtering by metadata and permissions | The search should be able to filter by source, project, date or permission within the query itself. Filtering a short hit list afterwards often leaves too few relevant passages. | Can tenants, projects or areas be kept cleanly apart? |
| Hybrid search | For part numbers, sections and names you need keyword search alongside vector search. Some databases include both; otherwise you run a second index and merge the result lists. | Is keyword search built in or run separately? |
| Data volume and memory | As a rough guide, 1 million chunks with 1,024 dimensions stored as 32-bit floats take about 4.1 billion bytes (3.8 GiB) of raw vector data, plus index structures, text and metadata. Many common index types keep this data in memory for fast search. | How many chunks do you expect in two years? |
| Licence | Check whether the licence allows self-hosting, modification and redistribution (for example Apache 2.0, MIT or the PostgreSQL licence) and whether key features are reserved for a commercial edition. Source-available licences often restrict running the software as a service for others. | Does the licence fit your use, including serving customers? |
| Deletion behaviour | Many vector indexes first only mark deleted entries as deleted (tombstones) and remove them physically during compaction. Snapshots and backups also keep deleted data until they expire. For erasure duties under Art. 17 GDPR, what counts is when the data is really gone. | When is data physically removed, and how long do backups live? |
Own illustration. Open-source examples of the deployment models, without ranking: ChromaDB (embedded or as a server) and pgvector (vector search as an extension of an existing PostgreSQL database). Memory maths: 1,000,000 × 1,024 × 4 bytes = 4,096,000,000 bytes. RAGSuite ships its own search index: ChromaDB, plus PostgreSQL for users, settings and history.
RAG evaluation: measure quality instead of guessing.
Every change to chunking, model or search can help or hurt. Without measuring, you only notice when users complain. So set up a fixed set of 50 to 100 real questions early, each with the expected source, and check the same metrics after every change.
| Level | Metric | The question behind it |
|---|---|---|
| Retrieval | Hit rate (recall@k) | Is the right passage among the top k hits? |
| Retrieval | Mean reciprocal rank (MRR) | How high up is the right passage? |
| Answer | Faithfulness | Is everything in the answer also in the sources? |
| Answer | Relevance | Does it answer the question asked? |
| Answer | Citation accuracy | Does the citation point to the right passage? |
| Operation | Feedback, latency, cost | Are users happy, and does it stay fast and affordable? |
In RAGSuite, every edition collects user feedback on answers. Detailed query tracing with timings for retrieval, reranking and model, plus Compare Models, are part of Enterprise.
Building a RAG system: develop it yourself or use a platform?
A prototype with a library and a few PDFs is quick to build. The effort is in operation: keeping sources in sync, respecting permissions, reindexing, providing an interface and monitoring quality.
| Task | Develop yourself | With RAGSuite |
|---|---|---|
| Connect sources | A separate integration per source | Upload, website crawl, SharePoint, Confluence, Google Drive, Notion, Slack, Gmail |
| Models | Integrate and maintain yourself | Choose the provider, including local with Ollama; switch embeddings with “Reindex now” |
| Search | Build index, parameters and reranker yourself | Vector index, Top-K, threshold, reranker by switch |
| Interface | Develop yourself | Chat, search, website widget, mobile app, REST API |
| Sources in answers | Implement yourself | Every answer with a source |
| Cost | Developer time and operation | Community Edition free, RAGSuite Engine from €39 a month (billed yearly, plus VAT) |
Before your RAG pipeline goes live.
Tick what is done. Nothing is stored.
Questions about the RAG pipeline.
What is a RAG pipeline?
A RAG pipeline is the chain of steps that makes documents searchable and passes the right passages to a language model for each question. It has two parts: indexing (read in, split, embed, store) and querying (search, rank, answer with a source).
What chunk size is right?
A good starting point is 300 to 800 tokens per chunk with 10 to 20% overlap, ideally split along headings. Smaller chunks hit more precisely; larger ones carry more context. Test with your own documents and questions.
Which vector database suits RAG?
There is no single best one. For a self-hosted RAG pipeline, what counts is the deployment model (embedded or a separate server), backup, filtering by metadata and permissions, hybrid search, the expected data volume, the licence and when deleted entries are really removed. For a single server, an embedded option is often enough; several applications and large data volumes point more towards a separate service.
What is hybrid search?
Hybrid search combines classic keyword search with vector search by meaning and merges the two result lists. It helps especially with part numbers, names, abbreviations and technical terms that pure vector search can miss.
What does reranking do?
A reranker scores the first 20 to 50 search hits again, more precisely, against the question and reorders them. Only the best five to eight passages then go to the language model. That improves accuracy but adds some time per request.
How do you measure the quality of a RAG pipeline?
With a fixed set of 50 to 100 real questions and expected sources. You measure retrieval (is the right passage found?) and the answer (is it grounded in the source, does it answer the question, is the citation correct?). Add user feedback and response time.
Should I build a RAG pipeline myself?
A prototype can be done in a few days. For production you add connectors, permissions, reindexing, an interface, logs and updates. A ready platform such as the free RAGSuite Community Edition saves most of that work.
Do I need to reindex after changing models?
For the embedding model, yes: vectors from different models are not comparable, so all documents have to be embedded again. For the chat model, no; you can switch it without reindexing.
The RAG pipeline, ready built. With a source for every answer.
Install the Community Edition for free, or let Sanjay show you how RAGSuite makes your documents searchable.