From Architecture to Output: Structural Origins of Hallucination in Large Language Models and the Amplifying Role of Data

Technical Summary: Structural Origins of Hallucination in Large Language Models

Problem Statement
Large language models (LLMs) consistently produce fluent, confident, yet factually incorrect outputs—a phenomenon known as hallucination. While existing taxonomies (e.g., Alansari and Luqman [1]; Huang et al. [2]) rigorously classify hallucinations by their output characteristics (intrinsic vs. extrinsic, faithfulness vs. factuality), they fail to identify the specific internal architectural mechanisms responsible for generating these errors. Current frameworks describe the symptom but do not locate the cause, leaving intervention strategies indirect and focused on managing outputs rather than addressing origins. This paper argues that hallucination is not a random surface phenomenon but a structural consequence of three specific architectural decisions that form a "compound failure system."

Methodology
The paper employs a mechanistic analysis of the Transformer architecture, tracing hallucination back to three core components:

1. Self-Attention: Analyzed as a mechanism that learns statistical co-occurrence rather than semantic truth.
2. Maximum Likelihood Estimation (MLE): Analyzed as a training objective that optimizes for statistical plausibility without factual constraints.
3. Autoregressive Decoding: Analyzed as a generation process subject to exposure bias, where errors cascade irreversibly.

To validate these theoretical claims, the authors conducted controlled empirical experiments using GPT-2 (a decoder-only autoregressive transformer). The experiments isolated each mechanism under specific prompting conditions:

• Self-Attention: Tested for co-occurrence misfires and positional bias using prompts designed to trigger learned associations in invalid contexts.
• MLE Objective: Tested for indifference to factual accuracy by comparing log-probability scores of common falsehoods versus verified truths.
• Autoregressive Decoding: Tested for cascade failures by introducing a single factual error in the prompt prefix and observing the propagation of that error through the generated sequence.

The paper also synthesizes findings from prior literature (e.g., Liu et al. [8] on attention bias; Lin et al. [9] on inverse scaling; Shumailov et al. [12] on model collapse) to demonstrate how dataset pathologies (long-tail deficiencies, training bias, synthetic pollution) interact with these architectural mechanisms.

Key Contributions
The paper makes three primary contributions:

1. Mechanism-to-Output Mapping: It establishes a direct mapping between the three architectural mechanisms and specific hallucination categories in the Alansari and Luqman [1] taxonomy:
   • Self-attention is identified as the source of intrinsic hallucination (entity confusion, fact misattribution, semantic drift) caused by the substitution of statistical proximity for semantic meaning.
   • MLE is identified as the source of extrinsic hallucination (imitative falsehoods) caused by rewarding statistically frequent patterns regardless of truth.
   • Autoregressive decoding is identified as the source of logical inconsistency caused by the permanent left-to-right commitment of tokens under exposure bias.
2. Reframing Dataset Pathologies: The paper argues that dataset pathologies (long-tail deficiencies, training bias, and synthetic pollution) do not independently cause hallucination. Instead, they act as amplifiers that exploit the pre-existing vulnerabilities of the three architectural mechanisms. For instance, long-tail deficiencies exploit the MLE objective's frequency-driven nature, while synthetic pollution feeds corrupted patterns into all three mechanisms simultaneously, leading to model collapse.
3. Diagnostic Limitation and Proposed Redesign: The paper identifies a critical gap in current diagnostic frameworks: classifying hallucination by output type alone prevents targeted intervention. It proposes a two-axis classification system that retains the output-type axis (intrinsic/extrinsic) and adds a mechanism-axis (attention, MLE, autoregressive) to pinpoint the specific architectural operator responsible for an error.

Results

• Empirical Validation: Experiments on GPT-2 confirmed the structural failure modes.
  • Self-Attention: In 13 of 15 prompts (86.7%), the model generated factually wrong completions based on strong co-occurrence patterns rather than semantic grounding. Attention weights concentrated on statistically dominant token pairs rather than semantically relevant context.
  • MLE: The model assigned higher or near-equivalent probability mass to common falsehoods compared to verified truths, confirming that the objective rewards frequency over accuracy.
  • Autoregressive Decoding: In 4 of 5 test pairs, a single wrong token in the prompt prefix caused the model to generate a fluent but factually broken sequence that compounded the initial error, demonstrating the cascade effect of exposure bias.
• Scaling Observations: The paper notes that scaling models under MLE does not resolve hallucination; rather, larger models learn the training distribution (including its falsehoods) more completely, potentially increasing the fidelity of extrinsic hallucinations (inverse scaling of truthfulness).

Significance
The paper claims that understanding hallucination as a structural consequence of architectural decisions is a necessary prerequisite for effective intervention. By shifting the focus from "what kind of hallucination occurred" to "which mechanism produced it," the field can move beyond indirect mitigation strategies (such as prompt engineering or retrieval augmentation) toward targeted architectural interventions. The authors argue that without identifying the specific architectural operator (attention, MLE, or decoding) responsible for an error, interventions remain blind to the root cause. This structural analysis provides a diagnostic framework that links specific failure modes to their architectural origins, offering a more complete picture of LLM reliability and a basis for future research into mechanism-level mitigation.