What is AI hallucination risk?

How hallucination occurs and why it persists

Large language models generate text by predicting the most statistically probable sequence of words given the input. They do not retrieve facts from a database; they generate text that resembles facts. When the model has strong training signal for a topic, it tends to produce accurate outputs. When the signal is weak — the topic is obscure, the training data was sparse, the question is at the boundary of the model’s knowledge — the model continues generating plausible text, but that text may not correspond to reality. The hallucination is characteristically confident: the model does not express uncertainty in the way a human would when uncertain. This is what makes hallucinations dangerous: they look like correct answers. They appear in text alongside genuinely accurate information, making it difficult for a non-expert reader to distinguish the accurate from the fabricated. The tendency persists across all current large language models because it is an inherent property of how they are built, not a bug that can be eliminated by retraining.

Human verification protocols for managing hallucination risk

Managing hallucination risk in a business context requires three elements. First, a clear policy on what AI outputs can be used for without verification and what requires verification before use. AI outputs used for internal brainstorming or drafting carry different risk than AI outputs used in client communications or regulatory submissions. The verification requirement should be proportionate to the consequence of an error. Second, a qualified human reviewer: verification is only meaningful if the reviewer has the expertise to identify errors. An AI-generated legal summary cannot be verified by someone without legal knowledge. The verification protocol must identify who is competent to verify each type of AI output. Third, an audit trail: documenting that verification occurred — who checked, when, what was checked — creates accountability and provides evidence of due diligence in the event that an error is later identified. The EU AI Act’s human oversight requirement for high-risk AI reflects these principles: genuine human oversight, not nominal review.