How to validate AI answers: hallucinations, who reviews and when

Why AI is wrong with such confidence

A language model does not “know” things: it generates the most likely text. When information is missing, it fills the gap with something that sounds right. It is like a brilliant intern who never says “I don’t know”: quick and self-assured, even when inventing the figure, the quote or the rule.

First, decide what an error costs

Not every answer deserves the same review. Classify each AI use by the damage a mistake would cause:

• Low: internal drafts, summaries, ideas. Sample-based review is enough.
• Medium: customer replies, reports, published content. A person validates before it goes out.
• High: decisions about people, legal, financial or health matters. The AI only proposes; the decision and its reasoning are human.

Five controls for reliable answers

1. Approved sources: have the AI answer from current, reviewed documents, and cite which one each answer comes from. That makes checking possible.
2. One person responsible for validation per use, with knowledge of the subject. Not simply whoever uses the tool: whoever knows if the answer is right.
3. A written validation criterion: a short list of what is always checked (figures, dates, names, quotes, rules) and what to do if something does not add up.
4. An error log: every detected failure is recorded and fixed at the source, not just in the answer.
5. A notice to the user: whoever receives the answer must know it comes from an AI and may be wrong.

Who validates, and when they must step in

Validation only works if the person can understand what they are validating and has time to do it. Three practical rules: when risk is high, it should be someone other than whoever generated the answer; they must have authority to reject it; and it should be defined in advance when intervention is mandatory (for example, any figure that will be communicated outside the company).

How to know it works

Each month, review a sample of answers and calculate three things: the share with errors, the average correction time and how many errors reached a customer or a decision. Also set a threshold: if errors exceed a certain level, use is paused until the cause is fixed.

Common mistakes

• Box-ticking review: approving without checking because “someone else looks at it”.
• Assuming RAG fixes it: connecting the AI to your documents reduces errors but does not remove them, and an outdated document produces outdated answers.
• Not measuring: without a monthly sample, nobody knows whether the control works.
• Treating every use the same: either everything is reviewed (unworkable) or nothing is.

What the regulation says

The AI Act requires effective human oversight for high-risk systems (Article 14) and appropriate levels of accuracy and robustness (Article 15). If an AI interacts with people, the transparency obligations of Article 50 may also apply. Even if your use is not high-risk, the same criteria are a sound basis.

Start here

1. List the AI uses in your company and classify them as low, medium or high risk.
2. For medium and high, name the person who validates.
3. Write the checklist (one page at most).
4. Schedule the monthly sample and record errors in a log.

Frequently asked questions

Can hallucinations be eliminated?

Not entirely. Good sources reduce them and validation detects them, but you should assume they will keep appearing.

Do all answers need validating?

No. Validation depends on risk: more control where an error costs more.

Who should validate?

Someone who knows the subject and has authority to reject the answer; in high risk, someone other than whoever generated it.

Informational content, not legal advice.

This gives you the judgment. If you want the finished document: Human Oversight Procedure.