Do LLMs Really Know What They Don't Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulness

This paper argues that LLM internal states primarily reflect the retrieval of parametric knowledge rather than the truthfulness of outputs, demonstrating that hallucinations driven by spurious statistical associations are mechanistically indistinguishable from factual recall, thereby limiting the effectiveness of standard detection methods.

The Gist

Here is an explanation of the paper "Do LLMs Really Know What They Don't Know?" using simple language and creative analogies.

The Big Question: Can AI Tell When It's Lying?

Imagine you have a very smart, well-read friend (the AI). You ask them a question. Sometimes they give you the right answer. Sometimes they make up a story that sounds perfect but is completely false (a "hallucination").

For a while, researchers thought: "Maybe our friend has a secret 'lie detector' inside their brain. Maybe when they are making things up, their brain waves look different than when they are telling the truth."

If this were true, we could build a tool to scan their brain and instantly know, "Ah, that answer is a lie!"

This paper says: No, that's not how it works.

The authors discovered that the AI's "brain waves" (internal states) don't tell us if the answer is true or false. Instead, they only tell us where the answer came from.

---

The Two Types of "Fake News"

The authors realized that not all hallucinations are created equal. They split them into two categories using a fun analogy: The Library vs. The Wild Guess.

1. Associated Hallucinations (The "Confident Librarian")

Imagine the AI is a librarian who has read millions of books.

• The Truth: You ask, "Where was Obama born?" The librarian pulls a book, reads "Honolulu," and says it.
• The Lie: You ask, "Where did Obama study?" The librarian knows the answer is Chicago. But, because "Obama" and "Chicago" appear together in so many books, the librarian gets confused. They confidently say, "Obama was born in Chicago!"

The Problem: In this case, the AI is using its real memory (the strong link between Obama and Chicago) to make a mistake.

• The Brain Signal: Because the AI is pulling from its real memory, its internal "brain waves" look exactly the same as when it tells the truth.
• The Result: The lie detector can't tell the difference. The AI is lying, but it feels just as "sure" to its own internal sensors as a true fact.

2. Unassociated Hallucinations (The "Wild Guesser")

Now, imagine you ask the librarian about a person nobody has ever heard of, like "Brenda Johnston."

• The Lie: The librarian has no idea who Brenda is. So, they just guess a random city, like "Portland."
• The Brain Signal: Since the AI has no memory of Brenda, it isn't pulling from its library. It's just guessing. Its internal "brain waves" look very different from when it's recalling facts. They look scattered and unsure.
• The Result: The lie detector works great here! It can easily spot, "Hey, this answer isn't coming from the library; it's just a random guess."

---

The Core Discovery: Memory vs. Truth

The paper's main finding is a bit of a bummer for AI safety:

The AI's internal signals tell us if it is "remembering" something, not if what it remembers is "true."

• If the AI is remembering a strong association (even if that association leads to a lie), its brain looks confident and "factual."
• If the AI is just guessing (because it has no memory), its brain looks shaky and "hallucinated."

The Analogy:
Think of the AI like a student taking a test.

• Associated Hallucination: The student knows the formula for a math problem but applies it to the wrong numbers. They are working hard, using their brain correctly, but the answer is wrong. A teacher looking at their scratch paper (internal states) sees a student working hard and thinks, "This looks like a real attempt!"
• Unassociated Hallucination: The student has no idea about the topic, so they scribble random numbers. The teacher looks at the scratch paper and sees chaos. "This is clearly a guess!"

The paper argues that current AI detectors are great at catching the "scribbling" (Unassociated Hallucinations) but terrible at catching the "wrong application of knowledge" (Associated Hallucinations).

Why Does This Matter?

1. We Can't Trust the "Lie Detector": If we rely only on the AI's internal signals to stop it from lying, we will fail. The AI will confidently lie about popular topics (like famous people or common facts) because those lies are built on real, strong memories.
2. The "Popular Subject" Trap: The paper found that these "confident lies" happen most often with popular subjects (like Obama or Paris) because the AI has seen them so many times. It's the opposite of what we might expect; we think the AI lies more about obscure things, but the dangerous lies happen when it's too confident about common things.
3. Teaching the AI to Say "I Don't Know": The authors tried to train the AI to say "I don't know" when it's unsure.
   • It worked great for the "Wild Guessers" (Unassociated). The AI learned to stop guessing.
   • It failed for the "Confident Librarians" (Associated). Because the AI felt so confident (since it was using real memory), it couldn't learn to stop. It kept lying confidently.

The Bottom Line

Large Language Models don't have a built-in "truth sensor." They have a "memory sensor."

• If they are pulling from memory, they feel confident, even if they are wrong.
• If they are guessing, they feel unsure.

To make AI safer, we can't just look at its internal brain waves. We need to build external fact-checkers (like a search engine or a human reviewer) to verify the answers, especially when the AI is talking about popular topics where it might be confidently wrong.

Technical Summary

Here is a detailed technical summary of the paper "Do LLMs Really Know What They Don't Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulness."

1. Problem Statement

Large Language Models (LLMs) frequently produce hallucinations (plausible but factually incorrect outputs). Recent research suggests that LLMs possess internal signals (e.g., hidden states, attention weights) that distinguish hallucinations from factual outputs, implying the models "know what they don't know."

However, this paper challenges that assumption. It argues that existing detection methods fail to distinguish between two fundamentally different types of hallucinations:

1. Knowledge Gaps: The model lacks the fact entirely.
2. Spurious Associations: The model relies on strong statistical correlations learned during pretraining that happen to be factually incorrect in a specific context.

The core problem is that current internal-state-based detection methods conflate truthfulness with knowledge recall. If a hallucination is generated via the same internal mechanism as a factual recall (e.g., leveraging a strong statistical shortcut), the internal states will look identical to a correct answer, rendering detection impossible.

2. Methodology

A. Taxonomy of Hallucinations

The authors propose a novel taxonomy categorizing model outputs based on their relationship to the subject entity and the internal mechanism used:

• Factual Associations (FAs): Factually correct outputs generated by recalling parametric knowledge.
• Associated Hallucinations (AHs): Factually incorrect outputs driven by strong, learned statistical associations involving the subject (spurious correlations).
• Unassociated Hallucinations (UHs): Factually incorrect outputs generated without reliance on strong subject-specific associations (often due to a lack of knowledge or random generation).

B. Dataset Construction & Causal Intervention

• Data: Constructed using Wikidata triples (Subject, Relation, Object) converted into cloze-style prompts.
• Models: LLaMA-3-8B and Mistral-7B-v0.3.
• Causal Intervention (Labeling): To distinguish AHs from UHs, the authors perform causal interventions on the model's internal states:
  • They block attention flow from subject tokens to the prediction token.
  • They measure the Jensen-Shannon (JS) Divergence between the original output distribution and the masked distribution.
  • High JS Divergence: Indicates the output relied heavily on the subject representation (categorized as AH).
  • Low JS Divergence: Indicates the output did not rely on the subject (categorized as UH).

C. Mechanistic Analysis

The authors analyze the internal computational pathways of these three categories:

1. Information Flow: Tracing how information propagates from early layers (subject encoding) to middle layers (attention flow) to upper layers (decoding).
2. Geometric Analysis: Examining the norm of subject representations, attention contribution norms, and the cosine similarity of last-token hidden states.
3. Detection & Tuning Experiments:
   • Detection: Training linear probes (white-box) and using scalar features (black-box) to classify FAs vs. AHs and FAs vs. UHs.
   • Refusal Tuning: Fine-tuning models to refuse answering AHs vs. UHs to test generalizability.

3. Key Contributions & Findings

A. Internal States Reflect Recall, Not Truth

The primary finding is that hidden states primarily encode whether the model is recalling parametric knowledge, not whether the output is true.

• AHs and FAs share identical internal mechanisms: Both rely on strong subject-to-token attention flows and similar subject representation norms. Consequently, their hidden-state geometries overlap significantly.
• UHs are distinct: They lack strong subject reliance, resulting in weaker information flow and distinct, clustered representations in the hidden space.

B. Failure of Current Detection Methods

• UH Detection: Highly effective. Since UHs have distinct geometries, detectors achieve high AUROC (≈0.86–0.93).
• AH Detection: Highly ineffective. Because AHs and FAs occupy the same representational space, detectors perform near random guessing (AUROC ≈0.48–0.69).
• Conclusion: Standard internal signals cannot distinguish a "confident hallucination" (AH) from a "confident fact" (FA).

C. Geometric Overlap and Subject Popularity

• Popularity Bias: AHs occur predominantly on high-popularity subjects (e.g., famous people) where statistical co-occurrences are strong in the training data. UHs occur on low-popularity subjects.
• Implication: The most dangerous hallucinations (those on popular topics) are the hardest to detect because they mimic the internal signature of facts.

D. Limits of Refusal Tuning

• UH Refusal: Models can successfully learn to refuse UHs (82% refusal rate) because these samples form a consistent, separable cluster.
• AH Refusal: Models struggle to learn to refuse AHs (33% refusal rate) because their representations are geometrically indistinguishable from FAs. The model cannot learn a decision boundary that separates AHs from FAs without also rejecting correct facts.

4. Significance and Implications

1. Rethinking "Self-Awareness": The paper refutes the idea that LLMs inherently "know what they don't know." They only "know" when they are not using their parametric knowledge. When they hallucinate via statistical shortcuts, they are internally "confident" just like when they are correct.
2. Detection Strategy Shift: Future hallucination detection cannot rely solely on internal state probing. It must distinguish between knowledge gaps (detectable) and spurious correlations (currently undetectable via internal states).
3. Evaluation Standards: Evaluation metrics must separate AHs and UHs. Reporting aggregate detection performance masks the critical failure to detect AHs.
4. Mitigation Pathways: Since internal tuning fails for AHs, the paper suggests that robust mitigation requires external verification mechanisms (e.g., retrieval-augmented generation, fact-checking modules) rather than relying on the model's internal confidence or hidden states.

Summary Table

Feature	Factual Associations (FA)	Associated Hallucinations (AH)	Unassociated Hallucinations (UH)
Truthfulness	True	False	False
Mechanism	Knowledge Recall	Spurious Statistical Shortcut	Knowledge Gap / Random
Subject Reliance	High	High	Low
Hidden State Geometry	Overlaps with AH	Overlaps with FA	Distinct / Clustered
Detectability	N/A (Ground Truth)	Very Low (AUROC ~0.5)	High (AUROC ~0.9)
Refusal Tuning	N/A	Fails (33% rate)	Succeeds (82% rate)

This work fundamentally shifts the understanding of LLM hallucinations from a binary "truth vs. lie" problem to a mechanistic distinction between knowledge retrieval and statistical association, highlighting a critical blind spot in current AI safety and interpretability research.