Intermediate

Why Models Hallucinate

Hallucination is not a bug in the traditional sense. It emerges from the fundamental mechanics of how LLMs are trained and how they generate text. Understanding the root causes tells you where the fixes have to live.

Root Cause 1: Token Prediction Is Not Fact Retrieval

The most fundamental cause of hallucination is a mismatch between what LLMs are trained to do and what users expect them to do. An LLM is trained to predict the most statistically likely next token given all the preceding tokens. It is not trained to retrieve stored facts and return them verbatim. The distinction matters enormously:

What users expect	What the model actually does
Look up the founding date of a company and return it	Predict what token comes next in a sequence that begins "The company was founded in"
Find the correct API method and call it	Generate tokens that look like an API call based on patterns seen in training data
Return a citation it has stored	Generate a sequence of tokens that looks like a citation in the appropriate format

When a model generates a plausible-looking date, citation, or API call, it is doing exactly what it was trained to do: producing the most statistically likely sequence. The problem is that "statistically likely given the surrounding context" and "factually correct" are not the same thing.

Root Cause 2: Training Data Gaps and Cutoffs

LLMs are trained on data collected up to a cutoff date. After that date, the model has no knowledge of events, product updates, regulatory changes, or any other new information. But the model does not know what it does not know: it will generate plausible-sounding answers about post-cutoff events with the same confidence as answers about well-documented historical events.

Training data gaps are not only temporal. They also appear for:

• Low-representation domains: Information that appears rarely in training data (specialized medical procedures, regional legal statutes, niche software libraries) is under-learned and more likely to be hallucinated.
• Conflicting training data: When the training corpus contains multiple conflicting facts about the same entity (because sources disagree or because the fact changed over time), the model may learn a blend of versions or produce inconsistent answers.
• One-to-many mappings: Names that map to multiple entities (a common name, an ambiguous acronym) are hallucination-prone because the model must pick one meaning from the probability distribution.

Root Cause 3: Confidence Miscalibration

A well-calibrated model would express uncertainty proportional to its actual accuracy: high confidence when it is likely right, low confidence when it is likely to be wrong. Most LLMs are miscalibrated in the direction of overconfidence. The model expresses certainty even when it should express doubt.

This is partly a training artifact: the model is rewarded for producing fluent, assertive text, and hedged, uncertain text is less common in the training corpus than confident assertions. After fine-tuning for helpfulness, models are further incentivized to give complete, confident-sounding answers rather than acknowledged non-answers.

The result is that the same linguistic markers of confidence, phrases like "The founding date was," "The correct method is," or "According to the study," appear in both correct and hallucinated outputs. Users cannot use the tone or certainty of the response as a signal of reliability.

Root Cause 4: Temperature and Sampling

At inference time, LLMs sample from a probability distribution over possible next tokens. The temperature parameter controls how broadly that sampling ranges: temperature 0 always picks the most likely token; higher temperatures pick from a wider distribution, producing more varied but less reliable outputs.

Higher temperature settings directly increase hallucination rate because they allow the model to sample less-likely tokens: tokens that may appear plausible given the context but are factually wrong. This trade-off is fundamental: creativity and diversity of output come at the cost of factual reliability.

Temperature Setting	Output character	Hallucination tendency	Best use
0.0-0.3	Deterministic, repetitive	Lowest	Factual tasks, data extraction, structured output
0.4-0.7	Balanced	Moderate	Analysis, Q&A, summaries
0.8-1.0+	Creative, varied	Highest	Creative writing, brainstorming, ideation

Root Cause 5: The "Lost in the Middle" Problem

Research has consistently shown that LLMs tend to attend most strongly to information at the beginning and end of their context window, and least strongly to information in the middle. This is sometimes called the "lost in the middle" problem: facts buried in a long context are more likely to be ignored or misrepresented than facts at the edges.

The practical consequence is that supplying a fact in context does not guarantee the model will use it. If the relevant information is in the middle of a 50,000-token context, the model may still generate a hallucinated answer rather than the in-context answer. This matters enormously for RAG systems where the retrieved document may not be at the beginning or end of the assembled context.

Root Cause 6: Instruction-Following vs. Knowledge Tension

LLMs are fine-tuned to follow instructions and be helpful. This creates a tension: if the user asks a question the model cannot accurately answer, the helpfulness objective pushes the model to generate a plausible answer anyway, rather than acknowledging the limits of its knowledge. The model has learned that "I don't know" is a socially less-acceptable response than a confident but wrong answer.

This is why explicit instructions to express uncertainty are so effective: they directly counteract the instruction-following bias toward confident answers. Lesson 5 covers how to write these instructions and how to verify they are working.