An AI hallucination is an answer that sounds convincing and is still wrong. The model invents a number, a source or a connection and presents it as confidently as any correct information. That's exactly what makes hallucinations dangerous: you can't tell from the answer.
For companies this isn't theory. In July 2025, the Cologne local court found that the sources in a lawyer's brief were "freely invented", apparently by an AI. So anyone using AI at work needs a clear picture of why hallucinations happen and what helps against them.
- An AI hallucinates when it presents something false convincingly as fact, such as an invented source, a wrong deadline or a feature that doesn't exist.
- That's because language models generate likely text and are rewarded in training for guessing rather than admitting gaps in their knowledge.
- Answers from your own documents with a visible source lower the risk considerably and make errors checkable, but they don't remove them entirely.
What are AI hallucinations?
People speak of a hallucination when a language model makes a false statement or falsely claims that a source supports a statement. Three forms are typical:
- Invented facts. A number, a date or a product feature that doesn't exist.
- Invented or wrong sources. A judgment, a section of law or an article that doesn't exist or says something else.
- Wrong summaries. The document is right, but the AI shortens or twists what it says.
The term is vivid but not quite accurate. The AI doesn't see things that aren't there. It produces text that sounds plausible without checking whether it's true.
In shortInvented facts, wrong sources, twisted summaries: convincingly worded.
Why does AI hallucinate?
A language model predicts, word by word, what is most likely to come next. It has no built-in knowledge of whether a statement is true. With common facts, the most likely continuation is usually right. With rare details, recent events or internal information, the model lacks the basis and fills the gap with something plausible.
Training adds to this. A 2025 research paper shows that the usual tests reward models for guessing rather than admitting uncertainty. Guessing in an exam sometimes earns points; saying "I don't know" never does. So models learn to give some answer rather than none.
For companies this means a general model can't reliably answer questions about your own contracts, products or processes. It has never seen that information.
In shortModels produce likely text and learn to guess rather than stay silent.
Examples from practice
The Cologne case shows the pattern. A custody brief quoted commentaries and law journals. One author didn't edit the commentary named, a margin number didn't exist, and a journal citation led to a completely different decision. The court pointed to the professional rules for lawyers.
In everyday business the errors are quieter, but no less harmful:
- In customer service, the AI gives a notice period that differs from the contract.
- It describes a product feature that is only planned.
- It mixes up the 2023 travel expense rule with the 2026 one.
All three have one thing in common: the answer sounds certain. Without a source, the error only shows once it has done damage.
In shortFrom the courtroom to customer service: without a source, the error shows too late.
What helps against hallucinations: answers from your documents
The most effective lever is not to let the model answer from memory. With RAG, retrieval-augmented generation, the system first searches your documents for matching passages and hands them to the model. The answer should rest on those alone. Our post What is RAG? explains how it works.
AI without your own sources
- Answers from what the model learned in training.
- Doesn't know your contracts, products or rules.
- Doesn't show where a statement comes from.
AI with answers from your documents
- First searches your documents for matching passages.
- Bases the answer on those passages.
- Shows the source, so anyone can check the statement.
The honest limit matters. In the study in the Journal of Empirical Legal Studies, even specialised tools connected to sources were wrong in 17 to 33 per cent of cases. They did better than general chatbots, but they weren't error-free. The model can misread a source, find the wrong passage or still fill the gap. The main gain from sources is therefore that errors become checkable.
In shortAnswers from your documents lower the risk and make errors checkable: they aren't error-free.
Six rules for everyday use
Technology lowers the risk; rules limit the damage. These six help in any company:
Every point ticked. Well prepared.
Training comes on top. Staff who know that even a well-worded AI answer can be wrong are more likely to open the source. That fits the AI literacy duty under Article 4.
In shortShow the source, allow "I don't know", maintain, narrow, test, double-check.
Questions for your AI vendor, including us
No vendor can promise that its AI never hallucinates. What counts is how well a system makes errors visible and how you can improve it. Ask:
- Does every answer show the passages it comes from?
- Does the AI say when your documents contain no answer?
- Can users report wrong answers, with a reason?
- Can you trace whether an error came from the search, the source or the model?
- Can you switch the model and compare results?
With RAGSuite, a knowledge engine for your data, every answer shows its source. Your team flags weak answers with a reason such as "outdated" or "wrong source", query tracing shows which step caused it, and models can be compared. That way the error rate falls over time. The quality loop shows how it all fits together.
This is general information. Always check important information against the source.
In shortAsk how a system makes errors visible, us included.
See for yourself how RAGSuite backs every answer with a source: 7 days free in RAGSuite Engine or in a call with Jürgen.
Frequently asked questions
What are AI hallucinations?
AI hallucinations are false statements that a language model presents convincingly as fact. They include invented numbers, sources or product features and wrong summaries of real documents. They're dangerous because they're worded just as confidently as correct answers. Without a visible source, they often only stand out once they've done damage.
Why does AI hallucinate?
Language models predict which text is most likely to come next, without checking whether it's true. When knowledge is missing, such as internal information, they fill the gap with something plausible. Research from 2025 also shows that training and tests reward guessing more than admitting that the model doesn't know.
How can you avoid AI hallucinations?
You can't avoid them completely, but you can reduce them considerably. What helps most is an AI that answers from your own documents and shows the source. Add maintained documents, clearly scoped assistants, tests with real questions, permission to say "I don't know" and human review for anything important.
Does RAG hallucinate too?
Yes, less often, but it happens. A 2025 study found error rates of 17 to 33 per cent in specialised tools connected to sources. The model can misread a source or find the wrong passage. The key advantage is that every answer shows its source, which makes errors checkable for anyone who reads it.
How do you spot an AI hallucination?
Most reliably by checking the source. If there's no source, the answer is unchecked. Warning signs are very precise figures without evidence, quotes you can't find and statements on recent or internal topics. Always check deadlines, amounts and legal questions in the original document before passing them on to anyone.
Sources
- Magesh et al.: Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Journal of Empirical Legal Studies 22 (2025): error rate
- Kalai et al.: Why Language Models Hallucinate (arXiv, September 2025): guessing is rewarded
- beck-aktuell: AG Köln, Beschluss vom 2.7.2025, 312 F 130/25: invented sources
- BSI: Große KI-Sprachmodelle: Chancen und Risiken für Industrie und Behörden: hallucinations, trust in text
Jürgen advises companies on introducing AI on their own infrastructure and leads RAGSuite at NITSAN. He writes about sovereignty, strategy and costs.