An open source LLM is a language model whose weights are freely available, so you can run it on your own hardware. Questions and documents stay in your network, and you are not tied to a model provider.
The choice is large and changes every month. Leaderboards help little, because they measure general tasks in English. What counts for your business is whether a model runs on your hardware, handles your documents well in German and fits your use legally.
- For language models, “open” usually means the weights are freely available. What you may do with them is set by the licence, and licences differ a lot.
- Hardware follows from model size. Rule of thumb: with 4-bit quantisation a model needs about half a gigabyte of GPU memory per billion parameters, plus headroom.
- Test two or three candidates with your own documents in German rather than going by leaderboards. On a platform with free model choice, you can switch later without starting over.
What is an open source LLM?
A large language model (LLM) consists of billions of learned numbers, the weights. With an open model, the maker provides these weights for download. You load the model onto your server and run it there without sending data to a provider.
The term “open source” is used loosely. In October 2024, the Open Source Initiative published a definition for open AI that requires information on training data and code as well as the weights. Many models considered open don't fully meet it. “Open weights” is often the more precise term.
Model via API
- Runs at the provider; question and passages go there.
- No GPU of your own needed.
- The provider sets price and model version.
Open model, self-hosted
- Runs on your servers; data stays in-house.
- Needs suitable hardware, usually a GPU.
- You decide version and when to switch.
Both have their place. Many companies use an open model for confidential data and a model via API for non-critical tasks. What matters is that your platform allows both.
Criterion 1: the licence
The licence decides whether you may use a model in your business at all. Check it before you invest time in tests.
| Licence type | What it usually allows | What to watch |
|---|---|---|
| Permissive (e.g. Apache 2.0, MIT) | commercial use, modification, distribution | include notice and licence texts |
| Maker’s own model licence | often commercial use with conditions | usage limits, prohibited uses, attribution duty |
| Research only | tests and research | no production use in a business |
With makers' own licences, read the list of prohibited uses and any thresholds above which a separate licence is required. Record which model runs in which version under which licence. That belongs in your documentation, like any other software.
Criterion 2: size and hardware
Model size is given in parameters, such as 8 or 70 billion. The larger, the more a model can do, and the more GPU memory it needs. Quantisation, storing the weights with fewer bits, cuts the need substantially, usually with little loss of quality.
| Model size | 16-bit (unquantised) | 4-bit (quantised) |
|---|---|---|
| about 8 bn parameters | about 16 GB | about 5 GB |
| about 30 bn parameters | about 60 GB | about 18 GB |
| about 70 bn parameters | about 140 GB | about 40 GB |
These are rules of thumb for the weights alone. On top comes memory for context, which grows with long documents and many concurrent users. Plan headroom and test under real load. For a RAG application, a mid-sized model is often enough, because it doesn't need to know everything: search supplies the matching passages.
Criteria 3 and 4: German and context length
Many models are trained mostly on English text. In German, spelling, technical terms and tone vary more than leaderboards suggest. So test with real questions and documents from your daily work: a policy, a contract, a product description.
Context length is how much text a model processes at once, measured in tokens. In a RAG application it must hold the rules, the question and the passages found. Bigger is not automatically better: a long context needs more memory and slows answers down.
All points ticked. Well prepared.
It sounds like effort, but it takes a few days if your platform lets you switch models without rebuilding.
Criteria 5 and 6: operations and updates
To run a model you need a runtime that loads it and answers requests. For getting started and small teams, a simple option such as Ollama is common. For many concurrent users, throughput-optimised servers such as vLLM are typical. Both offer an OpenAI-compatible interface.
Check the licence before you testChoose a runtime simple to start, throughput for many usersTest with your own data two or three candidatesPlan for switching roll in new versions without rebuilding
Open models move fast. What fits today may be outdated in a year. So plan updates as for other software: test the new version, compare it with the current model, then switch. That only works if your application isn't tied to one model, an idea our post on vendor lock-in explores.
In RAGSuite you choose the language model per project: locally via Ollama, vLLM or any OpenAI-compatible endpoint, or via API with your own key. Switching needs no reinstall, and with a local model RAGSuite also runs without any outside connection. The self-hosted AI page shows what that looks like on your servers; the free entry point is open source AI for business, the Community Edition.
Test open models with your own documents: with the free Community Edition or 7 days in RAGSuite Engine.
Frequently asked questions
What is an open source LLM?
An open source LLM is a large language model whose weights are freely available. You can download it and run it on your own servers, so no data goes to a model provider. What you may do with it is set by its licence. “Open weights” is often the more precise term.
Which open source LLM is best for business?
There is no single best model. Licence, available hardware, quality in German, context length and operations decide. Test two or three candidates with your own questions and documents. Leaderboards mostly measure general tasks in English and say little about your use.
What hardware do I need for a local LLM?
That depends on model size. As a rule of thumb, a model with 4-bit quantisation needs about half a gigabyte of GPU memory per billion parameters, plus headroom for context. A model with about 8 billion parameters runs on a single GPU; very large models need several.
May I use open source LLMs commercially?
With permissive licences such as Apache 2.0 or MIT, usually yes. Many models, however, are under the maker’s own licence with conditions, such as prohibited uses or usage limits. Some are released for research only. Check the licence before testing and record model and version.
Can I switch the model later?
Yes, if your application isn’t tied to one model. In RAGSuite you choose the language model per project and switch without reinstalling, whether locally via Ollama or vLLM or via API. That way you test new versions against the current model before switching.
Sources
- Open Source Initiative: The Open Source AI Definition 1.0 (October 2024): definition of open AI
- Apache License 2.0: permissive licence
- Ollama (GitHub): runtime for local models
- vLLM: documentation: high-throughput runtime
Mihaela takes customers from installation to their first production project. She writes about operations, connectors and the cost of running AI in your own data centre.