Local AI: install language models and run them in your company.
Local AI runs on your own hardware: no prompt goes to an AI provider, and after the download it also works offline. This guide shows which local AI models are suitable, what hardware you need, how to install a local LLM in an hour and what changes when a whole team uses it.
Free and open source · runs on your server · as of 3 November 2026
Key points
Local AI means the language model runs on your laptop or server. Questions, documents and answers stay in your network; after the download it also worksoffline .- Memory is the key hardware figure: about
0.6 GB per billion parameters at 4-bit quantisation, plus headroom. An 8-billion model runs on an 8 GB graphics card. - A first test takes an hour: install Ollama, download a model, ask questions. For a team you add a server, access control, updates and a platform for your documents.
- A local model does not know your documents. For it to answer from your knowledge you need RAG, for example with the free RAGSuite Community Edition.
What is local AI, and when is it worth it?
With an AI service on the internet, every question goes to the provider’s servers. With
It is worth it mainly in four cases. There are limits too, which you should know beforehand.
Confidential data
HR, customer and health data or trade secrets never leave your network.
Offline and isolated
Shop floor, lab or authority without internet: local AI still works there.
Predictable costs
No cost per request. You pay for hardware and operation, however much people ask.
Limits
The largest hosted models are stronger on hard tasks. Hardware, updates and operation are your job.
Local AI hardware: how much memory you need.
The key question is not the processor but the memory: the whole model has to fit in graphics memory (VRAM) or fast unified memory. Models are usually
The table shows rules of thumb. Allow headroom for the context window: the more text per question, the more memory.
These rules of thumb apply to one person. When many ask at once, every running request needs its own memory for its context (KV cache). With ten concurrent users at 8,000 tokens of context each, that cache (FP16) needs about twice the memory of an 8-billion-parameter model at 4 bits. The free LLM VRAM calculator works this out for your setup, including the embedding model for RAG.
| Model size | Memory at 4 bits | Suitable hardware | Good for |
|---|---|---|---|
| Small · 1 to 4 bn parameters | about 1 to 3 GB | Laptop, even without a GPU | Tests, short texts, classification |
| Medium · 7 to 14 bn | about 5 to 10 GB | GPU with 8 to 16 GB, or a machine with 16 to 32 GB unified memory | Questions about documents, summaries |
| Large · 27 to 34 bn | about 17 to 22 GB | GPU with 24 GB | Better quality for a team |
| Very large · 70 bn and more | from about 40 GB | Server with several GPUs, or 48 GB and more | High quality, many users |
Rule of thumb: parameters in billions × 0.6 GB at 4 bits, plus 10 to 20% headroom for context and runtime. At 8 bits roughly double. Actual figures depend on the model and context length.
Choosing local AI models: five criteria.
Open models come in many sizes and variants. Instead of a ranking that is out of date within weeks, these criteria help you choose. Then test two or three candidates with your real questions.
Licence
Does the licence allow commercial use? Some models limit user numbers or purpose.
German quality
Not every model writes good German. Check terminology and tone with your own texts.
Size and speed
Larger models answer better but more slowly and need more memory. Choose to fit your hardware.
Context length
How much text fits in one request? Questions about documents need room for the passages found.
Embedding model
For searching documents you also need an embedding model, also local.
Install local AI: a test in one hour.
For a first test, use Ollama, a free open-source tool that downloads local LLMs and serves them through an interface. For macOS and Windows there is an installer on ollama.com; on Linux one command is enough.
Replace the placeholder with a model from the Ollama library that fits your hardware and licence requirements. Newer Ollama versions also offer cloud models and web search; for strictly local use, use local models only and turn online features off (see the Ollama documentation).
$ curl -fsSL https://ollama.com/install.sh | sh $ ollama pull <model> # download a model $ ollama run <model> # chat in the terminal ✔ API http://localhost:11434
Install
Install Ollama from ollama.com. The interface then runs locally on port 11434.
Download a model
Use ollama pull to download a model that fits your memory.
Test
Chat with ollama run. Ask ten typical questions and compare the answers.
Disconnect
Disconnect from the internet and ask again. If it keeps working, it really runs locally.
Running local AI in a company: what changes.
A laptop test shows whether a model is good enough. For a team it becomes a small infrastructure project: a server in your network, several users at once, access rights, logs and updates.
Above all, the model lacks your knowledge. For it to answer from manuals, contracts and policies, you need a platform that reads documents in, searches them and shows answers with sources.
| Topic | Laptop test | Team operation |
|---|---|---|
| Hardware | Existing machine | Server with a GPU in your network |
| Users | One person | Many at once, with sign-in |
| Knowledge | Only the model | Your documents via RAG, with sources |
| Access | Local only | Internal network only, rights per project |
| Operation | Manual | Updates, monitoring, backups, owners |
How to check that your local AI really sends nothing out.
“Runs locally” is a claim you can check yourself, and for your data protection officer and works council you should. Five steps give you solid evidence, and the result belongs in your documentation, for example in the data protection impact assessment.
The RAGSuite Community Edition sends no telemetry to the vendor. The Enterprise licence is also activated offline with a locally checked key, with no licence server.
Disconnect
Disconnect the internet and ask the same test questions. Everything must keep working, including document search.
Log connections
With the connection back on, log outgoing traffic at the firewall or proxy while the team works normally.
Check DNS
Which names does the server resolve? Expect only internal services and, if allowed, your update mirror.
Turn off online features
Disable cloud models, web search and automatic updates of the runtime; load models from an internal mirror.
Document it
Record date, version, test questions and the log. Repeat after every update.
No servers of your own? The alternative based in Germany.
Not every company wants to buy and run GPUs. With the RAGSuite Engine you get the same platform with a data centre in Germany, including storage and credits, from €39 a month billed yearly, plus VAT. If you want to self-host later, you move to the Community Edition or Enterprise.
Before local AI goes into production.
Tick what is done. Nothing is stored.
Questions about local AI.
What is local AI?
Local AI means a language model runs on your own hardware, on a laptop or a server in your company network. Questions and answers do not leave your network, and once the model is downloaded the AI also works offline.
What hardware do I need for local AI?
Memory is what matters. As a rule of thumb, a model quantised to 4 bits needs about 0.6 GB per billion parameters, plus headroom for the context. An 8-billion-parameter model therefore runs on an 8 GB graphics card; a 70-billion model needs 40 GB or more.
Does local AI run without a graphics card?
Yes, small models also run on the CPU, but much more slowly. That is fine for tests. As soon as several people ask at the same time, you need a GPU or a machine with fast unified memory.
Is local AI as good as large cloud models?
For general, hard tasks, the largest hosted models are usually stronger. For questions about your own documents, a good local model is often enough, because the answer is built from the passages found. Test with your real questions.
How do I install a local LLM?
The quickest way is Ollama: install it, download a model with ollama pull and start it with ollama run. For team use you add a server, access rules, updates and a platform for your documents.
Is local AI free?
Many models and tools are free and open source, including the RAGSuite Community Edition. Costs come from hardware, power and operation. Check each model’s licence to see whether it allows commercial use.
What is the difference between local and private AI?
Local AI describes where the model runs: on your hardware. Private AI describes that your data does not end up with the provider and is not used for training. Local AI is the strictest form of private AI.
What is a self-hosted LLM?
A self-hosted LLM is a language model you run on your own servers or in your own cloud environment instead of through an AI provider’s interface. You decide where data is processed, who has access and when updates happen.
How much memory does local AI need for several users?
More than for one person, because every concurrent request needs its own memory for its context. With ten users that share can be larger than the model itself. The free LLM VRAM calculator shows the need for model, context, user count and embedding model.
Local AI with your documents, with no data going out.
Install the Community Edition, or let Mihaela show you the way from test to production.