Understanding New-Knowledge-Induced Factual Hallucinations in LLMs: Analysis and Interpretation

1. Problem Statement

Large Language Models (LLMs) acquire factual knowledge during pre-training but often require Supervised Fine-Tuning (SFT) to learn new, specific knowledge for downstream tasks. However, prior research indicates that introducing new knowledge during fine-tuning can induce factual hallucinations, causing the model to generate incorrect outputs even on previously known information.

While the existence of this phenomenon is established, the paper identifies critical gaps in understanding:

• Granularity: How do hallucinations manifest across different knowledge types (e.g., birth year vs. major) and task types (e.g., direct QA vs. complex reasoning)?
• Mechanism: What are the underlying neural mechanisms (e.g., attention shifts) driving these hallucinations?
• Propagation: How do hallucinations spread from new knowledge to unrelated tasks or lexically similar contexts?
• Mitigation: Can specific training interventions restore model stability without discarding the new knowledge?

2. Methodology

A. Dataset Construction: Biography-Reasoning

To overcome the difficulty of controlling "known" vs. "unknown" knowledge in real-world datasets, the authors constructed a synthetic, controlled dataset called Biography-Reasoning.

• Structure: The dataset centers on synthetic individuals with four attributes (Knowledge Types): Birth Year, Death Year, Major, and University.
• Tasks: It includes two main task categories:
  1. Knowledge QA: Direct questions about attributes (e.g., "What is X's major?").
  2. Knowledge Reasoning: Three complex reasoning types requiring Chain-of-Thought (CoT):
     • Single Reasoning: Extracting an attribute and performing simple logic.
     • Comparative Reasoning: Comparing attributes between two entities.
     • Novel Reasoning: Applying new definitions (e.g., mathematical operations on extracted data).
• Control: The dataset allows precise manipulation of the proportion of Known (seen during pre-training/initial CPT) vs. Unknown (newly introduced) knowledge within specific types and tasks.

B. Experimental Setup

• Models: Primary experiments used Qwen2.5-1.5B, with validation on Llama3.2-1B, Qwen3-8B, and Qwen2.5-32B.
• Training Protocol:
  1. Continual Pre-Training (CPT): Models are exposed to a subset of "Known" data to establish a baseline.
  2. SFT: Models are fine-tuned on mixed datasets where specific knowledge types are replaced with "Unknown" data.
  3. Evaluation: Performance is measured on Same-Type, Different-Type, and Out-of-Distribution (OOD) Wiki test sets using Exact Match (EM).

C. Interpretability Analysis

The authors analyzed attention mechanisms to understand why hallucinations occur. They specifically measured attention scores on key entities (person names) in the input question during the generation of the answer.

D. Proposed Intervention: KnownPatch

Based on their findings, the authors propose KnownPatch, a training strategy where a small amount of Known Knowledge is injected during the late stages of training (after the model has learned the new knowledge) to restore attention patterns.

3. Key Findings and Results

A. Manifestation of Hallucinations

• Severity: Learning new knowledge significantly degrades performance on tasks involving that specific knowledge type (Same-Type QA).
• Propagation: Hallucinations are not isolated. They propagate to:
  • Different Knowledge Types: Performance drops on unrelated attributes (e.g., learning new "Majors" hurts "Birth Year" QA).
  • OOD Tasks: Performance degrades on real-world Wiki datasets, indicating a general loss of factual grounding.
• The "All-Unknown" Effect: The most critical finding is that familiarity within a knowledge type is more important than the global proportion of new data. If a specific knowledge type (e.g., "Major") consists entirely of new data, even a small amount of such data triggers severe hallucinations. Conversely, if a type is a mix of known and unknown, the hallucination risk is lower.

B. Mechanism: Attention Shift

• Attention Decay: Learning new knowledge causes the model to weaken its attention to key entities (e.g., the person's name) in the input question.
• Context Over-reliance: As attention to the entity drops, the model over-relies on surrounding context tokens, leading to the generation of plausible but fabricated facts.
• Correlation: There is a strong negative correlation between the drop in entity attention scores and the increase in hallucination rates.

C. Propagation via Lexical Similarity

• The paper demonstrates that hallucinations propagate across tasks primarily through lexical similarity (token overlap), not semantic similarity.
• Tasks with input contexts that share many tokens with the "Unknown" training data are more susceptible to hallucinations, even if the semantic meaning of the task is different.

D. Effectiveness of KnownPatch

• Restoration: Injecting a small amount of known knowledge (5%–20%) at the end of training effectively restores attention to key entities.
• Performance Recovery: This intervention significantly mitigates hallucinations, restoring performance on both in-domain and OOD test sets to near-baseline levels.
• Generalization: The method works even if the injected known data does not cover the specific unknown knowledge types, suggesting it acts as a regularizer for the model's attention mechanism rather than simple knowledge replay.

4. Key Contributions

1. Fine-Grained Analysis: The first study to disentangle hallucination effects by knowledge type and task complexity, revealing that "fully unknown" types are the primary trigger for severe hallucinations.
2. Mechanism Interpretability: Identifies attention shift away from key entities as the root cause of new-knowledge-induced hallucinations.
3. Propagation Driver: Establishes lexical similarity as the primary vector for cross-task hallucination spread, challenging the assumption that semantic relatedness is the main driver.
4. Practical Solution (KnownPatch): Proposes a simple, low-cost training intervention (late-stage injection of known data) that effectively stabilizes models learning new knowledge without requiring complex architectural changes or data filtering.

5. Significance

This work provides a crucial theoretical and practical framework for understanding the "catastrophic forgetting" or "factual instability" that occurs when LLMs learn new facts.

• For Researchers: It shifts the focus from data volume to data composition and attention dynamics, offering a new lens for analyzing model failure modes.
• For Practitioners: It suggests that when fine-tuning models on new domains, practitioners should ensure that some known examples of the target domain remain in the training set (or be re-introduced later) to prevent the model from losing its grounding. The KnownPatch method offers a lightweight, effective strategy to deploy models that are both knowledgeable and factually reliable.