If you want to use AI with your own data, in most cases you do not need to train it. For questions about policies, manuals or products, it is usually enough to give the language model the right passages with each question. This approach is called retrieval-augmented generation, or RAG.
Training and fine-tuning still have their place, for example when a model should learn a particular style or technical language. This post explains the three routes in plain words, compares cost, freshness and data protection, and shows when each route fits.
- Training a model from scratch is too expensive for almost every company. Fine-tuning and RAG are the realistic options.
- Fine-tuning changes the model and suits style, format and technical language. New knowledge only gets in through more training.
- RAG leaves the model unchanged and gives it the right passages with each question. New knowledge is available once it is read in, and every answer can name its source.
Three routes to an AI that uses your knowledge
A language model learns its knowledge during training from large amounts of text. What comes after, such as your policies, products and processes, it does not know. There are three routes to change that:
| Route | What happens | Typical use |
|---|---|---|
| Training from scratch | a new model is learned from very large amounts of data | language model providers |
| Fine-tuning | an existing model is trained further with your own examples | style, format, technical language, a specific task |
| RAG | the model stays unchanged; the right passages are supplied with each question | questions about documents, policies, products |
Training from scratch needs enormous amounts of data and computing power and is hardly an option for companies outside the AI industry. In practice, the choice is between fine-tuning and RAG, or a combination of both.
Training and fine-tuning: changing the model
With fine-tuning you train an existing model further with your own examples, such as question-and-answer pairs from your support. Methods such as LoRA keep the original weights frozen and train only a small number of added parameters, which makes further training much cheaper than full training.
Fine-tuning is good at teaching a model a behaviour: a tone, an output format, technical terms or a narrowly defined task. It is less good as a store for facts. The model does not reliably reproduce what it learned word for word, cannot reliably say where a statement comes from, and every change in knowledge needs another training round.
Fine-tuning
- Changes the model’s weights.
- New knowledge needs more training.
- Answers give no reliable source.
Well suited for
- Tone and output format.
- Technical language and fixed tasks.
- Combining with RAG to get both.
RAG: supply knowledge instead of training
With RAG, the language model stays unchanged. Your documents are read in and put into a search index. When a question comes in, the system finds the right passages and passes them to the model. The model writes the answer from them and can point to the source.
The advantage: new knowledge is available as soon as the document is read in, with no training. Deleted documents drop out of the index. And because the answer comes from specific passages, it can be checked in the original. Errors are not ruled out, for example if the search finds the wrong passage; more on that in the post on AI hallucinations.
Read in documents, wiki, stores into the search indexSearch find the right passages for each questionAnswer the model writes from the passagesCite the source is shown alongside
How a RAG system is built in detail is explained in the post on the RAG system.
Comparison: cost, freshness, data protection
For most business uses, three questions decide: what does it cost, how current does the knowledge stay, and how do you handle personal data?
| Criterion | Fine-tuning | RAG |
|---|---|---|
| Cost | training, data preparation, retraining when things change | reading in and search index; no training |
| Freshness | as of the last training | as of the last import |
| Sources | no reliable source given | answer with source possible |
| Permissions | knowledge sits in the model, the same for everyone | content can be separated by project |
| Deletion | targeted removal from the model is difficult | remove the document from the index |
The difference is clearest for data protection. In its Opinion 28/2024, the European Data Protection Board stated that AI models trained on personal data are not automatically anonymous. Anyone who trains personal data into a model must therefore also be able to answer questions such as access and erasure. With RAG, the data stays in your documents and the search index, where you manage and delete it.
When each route fits
The rule of thumb: if it is about knowledge, content that changes and should be cited, choose RAG. If it is about behaviour, how a model writes or performs a task, fine-tuning can help. Many teams combine both: an adapted model that answers from the passages of a RAG engine.
All points ticked. Well prepared.
To get started, we almost always recommend RAG: ready faster, no training data needed and answers that can be checked.
AI with your own data in RAGSuite
RAGSuite is a RAG engine. It connects your sources, from upload and website to SharePoint, Confluence, Google Drive, Notion and Slack, and answers questions with sources. RAGSuite does not use your data to train any model. You choose the language model per project, hosted or local via Ollama.
You can run RAGSuite yourself with the free Community Edition, or use RAGSuite Engine in a data centre in Germany. The AI knowledge base page shows more on the knowledge base, and the RAG Engine page how the engine works.
See for yourself how RAGSuite uses your knowledge without training a model: free for 7 days with the RAGSuite Engine Starter plan or in a call with Mihaela.
Frequently asked questions
Can you train an AI on your own data?
Yes, through fine-tuning, that is, training an existing model further with your own examples. For questions about documents, policies or products, though, it is rarely the best route. RAG is usually enough: the model stays unchanged and receives the right passages from your documents with each question.
What is the difference between RAG and fine-tuning?
Fine-tuning changes the model itself and suits style, format and technical language. RAG leaves the model unchanged and supplies passages from your documents with each question. With RAG, new knowledge is available after import, and answers can name their source.
What does it cost to train an AI on your own data?
Fine-tuning costs computing time, but above all work on good training examples and retraining whenever things change. Training from scratch is not economical for most companies. RAG needs no training; costs arise for import, the search index and using the language model.
Is RAG better for data protection than training?
Usually, yes. With RAG, personal data stays in your documents and the search index, where you manage and delete it. In a trained model it can hardly be removed in a targeted way. The European Data Protection Board does not consider trained models automatically anonymous.
Does RAGSuite train models on my data?
No. RAGSuite is a RAG engine: your data is read in and searched, but not used to train a model. In RAGSuite Engine this also applies to Microsoft, which handles AI processing in the EU/EFTA: Microsoft does not train models on your data. To detect misuse, Microsoft checks flagged requests automatically and, where needed, through staff in the EEA; this data stays in Germany. Self-hosted or with your own API key, your model provider’s contract governs whether it uses data.
Sources
- Lewis et al. (2020): Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: foundational RAG paper
- Hu et al. (2021): LoRA, Low-Rank Adaptation of Large Language Models: cheaper fine-tuning
- EDPB: Opinion 28/2024 on AI models (December 2024): personal data in AI models
Mihaela takes customers from installation to their first production project. She writes about operations, connectors and the cost of running AI in your own data centre.