Local LLM with Ollama: RAG on your own documents, step by step.
A local LLM on your own data means a language model runs on your own computer or server and answers questions from your documents. Ollama runs the model. For it to answer questions from your documents, it needs a search that finds the right passages for each question and passes them along: retrieval-augmented generation, or RAG. This guide shows how to connect Ollama to the free RAGSuite Community Edition.
Free and open source · runs on your server · as of 5 November 2026
Key points
- Ollama serves local language models through a standardised interface, locally at
http://localhost:11434/v1. - RAGSuite Community Edition handles the RAG part: reading in documents, searching them and showing answers with sources. You select Ollama in the model settings as the provider “Custom LLM / Ollama”.
- For no data to leave your network, both the chat model and the embedding model must run locally. If you use a hosted provider for either, questions and passages go there.
- The most common mistakes: Ollama cannot be reached from a Docker container, the context window is too small, or reindexing is missing after the embedding model changes.
What Ollama and RAG do together.
Ollama alone does not know your documents. Only together with a RAG platform does a local model answer questions from your knowledge. The jobs are clearly split:
Ollama: the model
Ollama downloads open language models and runs them on your hardware. It writes answers but does not search a document collection.
For each question, the right passages are found in your documents and passed to the model, so it answers from your sources.
RAG system →Connect sources, build the search index, chat and search with sources. Open source under Apache 2.0.
Open-source AI for business →Together: all local
When model, index and platform run on your own infrastructure, no request leaves your network. RAGSuite sends no telemetry.
Prerequisites for RAG with Ollama.
| Component | What you need | Note |
|---|---|---|
| RAGSuite CLI | Node 18+ (20/22 LTS recommended), npm, Git 2.30+ | macOS, Linux or Windows with WSL2/Git Bash |
| Native mode (default) | Python 3.14, Yarn 1.22+, PostgreSQL 15+, Redis 7+ | Check with ragsuite doctor |
| Docker mode (optional) | Docker with Compose v2 | Select with ragsuite init --docker |
| Ollama | Ollama installed, one chat model and one embedding model pulled | Install as described on ollama.com |
| Hardware | Enough memory for the chosen model; a GPU speeds things up a lot | Small models also run on the CPU, but more slowly |
Step by step: pull the models and install RAGSuite.
First pull two models into Ollama: a chat model for answers and an embedding model for the search index. Replace the placeholders with models from the Ollama library whose licence allows business use.
Then install and start RAGSuite through the public CLI. Native mode is the default; Docker is optional.
$ ollama pull <chat-model> # model for answers $ ollama pull <embedding-model> # model for the search index $ ollama list # check pulled models
$ npm install -g @ragsuite/ragsuite@latest $ ragsuite init # default: native $ ragsuite start ✔ web http://localhost:9191 ✔ api http://localhost:9090
Connect Ollama to RAGSuite.
After installation, you connect the two in the web interface. The model settings look roughly like this; your model names appear where the placeholders are.
Create a project
Open http://localhost:9191 and create a project.
Choose the provider
In Model Settings, choose the provider “Custom LLM / Ollama” and add the address of your Ollama server, locally usually port 11434 with the path /v1. The documentation describes the exact field.
Assign the models
Choose your chat model and your embedding model. Only if both run locally do no passages leave your network.
Check the connection
Save the settings and ask a test question. If the chat answers with a source, Ollama is connected.
Connect sources
Upload documents or connect SharePoint, Confluence, Google Drive, Notion or Slack. The chat then answers questions from your documents, with sources.
You find the exact field names and further settings in the documentation at docs.ragsuite.de.
Common mistakes with RAG and Ollama, and how to fix them.
| Problem | Cause | Fix |
|---|---|---|
| RAGSuite cannot reach Ollama (Docker mode) | Inside the container, localhost points to the container itself, not to your server. | Use the host’s address, with Docker Desktop for example host.docker.internal (on Linux also add host-gateway), and bind Ollama to a reachable address. |
| Ollama cannot be reached from another machine | By default Ollama listens only on 127.0.0.1, port 11434. | Set another address with OLLAMA_HOST, for example 0.0.0.0:11434, and open the port only on the internal network. |
| Answers skip parts of the passages found | The model’s context window is too small for the question and passages. | Increase the context with OLLAMA_CONTEXT_LENGTH or retrieve fewer passages per question (Top K Results). |
| Hits are missing after a model change | The search index was built with a different embedding model. | Reindex the sources (“Reindex now” in the model settings). |
| Answers are very slow | The model runs on the CPU or is too large for the hardware. | Choose a smaller model or use a GPU. |
| Answers in poor German or the wrong language | Not every model handles every language equally well. | Choose a model with good quality in your language and test with your own questions. |
From test to production.
A test on a laptop shows whether model and documents fit together. For team use, plan these points:
Own server
Ollama and RAGSuite on a server in the company network, with a GPU if several people ask at the same time.
Multiple users
The Community Edition has unlimited users and projects. Enterprise adds roles, SSO and a full audit log.
Updates
Update RAGSuite with ragsuite update --restart; database, .env and keys are kept.
Check quality
Test with real questions and check the sources. A small local model often answers more briefly than a large hosted model.
Before you use RAG with Ollama in production.
Tick what is done. Nothing is stored.
Questions about RAG with Ollama.
What is Ollama?
Ollama is an open-source tool for downloading and running language models on your own hardware. It serves the models through a local, standardised interface that many tools can address.
How do I use a local LLM with my own data?
You need two building blocks: a local language model, for example through Ollama, and a RAG platform that reads in your documents, builds a search index and supplies the right passages for each question. With the free RAGSuite Community Edition, you load a chat model and an embedding model in Ollama, install RAGSuite and select Ollama as the provider in the model settings. If both models run locally, no data leaves your network.
Can Ollama do RAG on its own?
No. Ollama runs the model but does not search a document collection. For RAG you need a platform that reads in documents, builds a search index and supplies the right passages for each question. RAGSuite Community Edition does that.
Is RAG with Ollama free?
Ollama and RAGSuite Community Edition are free and open source. Costs arise for hardware, power and operations. If you do not want to run your own servers, you can use RAGSuite Engine, from €39 a month billed yearly, plus VAT.
Which model should I use?
That depends on language, hardware and task. Look for good quality in your language, a licence that allows business use and a size that fits your memory. You can switch the model in RAGSuite later.
Does data leave my network?
Not if the platform, search index, chat model and embedding model run on your own infrastructure. RAGSuite sends no telemetry. If you choose a hosted provider for either model, questions and passages go to that provider.
Does it work without internet?
You need internet for installation: npm for RAGSuite, and Ollama to pull the models. After that, models and platform can run without any outside connection. For fully isolated environments we support you as part of Enterprise.
Local AI with your documents, with no data going out.
Install the Community Edition, or let Mihaela show you the way from test to production.