Detecting Hallucinations

The central challenge of hallucination detection is that hallucinated outputs are indistinguishable from correct outputs by surface features alone. This lesson covers the four detection methods that actually work, and when to apply each one.

The Detection Challenge

Hallucinations are hard to detect for the same reason they are dangerous: they look exactly like correct outputs. The fluency, the authoritative tone, the plausible structure: these are features of the model's generation process, not signals of accuracy. You cannot read a hallucinated paragraph and know it is wrong without external knowledge or verification.

This means detection methods must be external to the model's primary output. You need either a reference source to compare against, multiple model outputs to cross-check, a second model to evaluate the first, or human expert review. Each approach has different cost and coverage characteristics.

Method 1: Cross-Reference Checking

The most reliable detection method for closed-domain tasks is checking the output against a source document. If you provided a document in context and the model summarized or answered questions about it, every claim in the output should be traceable back to the source.

How it works: Parse the model's output into individual claims, then check each claim against the source material. This can be done by a human reviewer, by a second LLM call ("Does the source document support this claim?"), or by a specialized faithfulness evaluation tool.

When it works: Closed-domain tasks where a reference source exists (document summarization, Q&A over a knowledge base, contract analysis, code explanation from a specific codebase).

When it fails: Open-domain tasks where there is no reference source; extrinsic hallucinations that add information the source does not contain (the source cannot confirm or deny).

Method 2: Consistency Sampling

Consistency sampling exploits a property of hallucinated claims: they tend to be less stable across multiple generations than correct claims. If you ask the same question multiple times (with temperature > 0 to allow variation), a correct answer should appear consistently, while a hallucinated answer may vary.

How it works: Send the same prompt N times (typically N=3-5) and compare the outputs. Claims that appear identically in all N outputs are more likely to be correct. Claims that vary across outputs are candidates for further review.

When it works: Factual claims about entities the model has knowledge of. Questions where a single correct answer exists.

When it fails: Consistently hallucinated facts: if the model always generates the same wrong answer (because that wrong answer is the most probable token sequence given training), consistency does not catch it. Also fails on open-ended questions where variation is expected and correct.

Method 3: LLM-as-Judge Evaluation

A second LLM call, given the original output and (optionally) a reference or rubric, evaluates whether the output contains hallucinations. This is increasingly the standard evaluation pattern for production systems.

How it works: Construct a verification prompt such as: "Here is a response to the query. Evaluate each factual claim in the response. For each claim, indicate whether it is: (a) supported by the provided context, (b) not mentioned in the context, or (c) contradicted by the context. Return structured JSON." Send to a judge model (often a different model than the one that generated the output, or the same model with a more reliable system prompt).

When it works: Closed-domain faithfulness evaluation; structured output where claims can be enumerated. Works well when paired with RAG (can check claims against retrieved chunks).

When it fails: Open-domain factual claims where the judge model also lacks the correct information; reasoning hallucinations where the judge model makes the same logical error; costs become significant at high volume.

Method 4: Structural and Executable Verification

For code and structured data outputs, you can verify mechanically rather than semantically. This is the most reliable form of hallucination detection available, when applicable.

• Code: Run the generated code in a sandbox. A method that does not exist throws an error. A wrong parameter raises a TypeError. Static analysis catches undefined references before execution.
• Citations: Check that cited URLs resolve and return relevant content. Check that DOIs correspond to real papers. Check that cited cases appear in legal databases.
• Structured data: Validate against a schema. If the model claims a date is "2031-02-30" (February 30 does not exist), a date validator catches it immediately.

Decision Table: Which Method for Which Task