Reality Monitoring in Large Language Models: Self-Knowledge That Transforms with Conversation Memory

Technical Summary: Reality Monitoring in Large Language Models

Problem Statement
As Large Language Models (LLMs) transition from static tools to autonomous agents operating in high-stakes, multi-turn environments (e.g., clinical support, legal analysis), their ability to distinguish between internally generated content and externally provided information becomes a critical safety issue. In human cognitive science, this capacity is termed reality monitoring (RM): the ability to determine the origin of a memory (externally perceived vs. internally imagined). Failures in human RM are linked to hallucinations, delusions, and confabulation. However, it remains untested whether LLMs possess this capacity. Current benchmarks often fail to isolate conversational memory as a variable, and existing metacognitive assessments focus on knowledge accuracy rather than source attribution. A model that cannot attribute its own output to itself may treat its own confabulations as user-provided facts, leading to safety failures rather than mere performance errors.

Methodology
The authors adapted the standard human RM word-pair paradigm to assess source monitoring in six LLMs: Gemma 3 (12B and 27B, with and without Quantization-Aware Training), Llama 3.3 (70B), and a sparse mixture-of-experts model (Llama 4 16×17B). The task required models to encode word pairs where the second word was either externally provided (perceived) or internally generated (imagined), and subsequently attribute the source of that word.

Two experiments manipulated conversational memory architecture:

1. Experiment 1 (Single-Trial Reality Monitoring): Compared two memory structures:
   • Single-Turn: All subtasks (completion, rating, source judgment, confidence) occurred within one interaction.
   • Trial-Chain: Subtasks were distributed across sequential conversational turns, increasing the context window load.
2. Experiment 2 (Episodic Reality Monitoring): Introduced a delayed source-test phase. Encoding trials (20 or 40 total) were completed as a block before a separate source-attribution phase. This design crossed set size (20 vs. 40) with corrective feedback (present vs. absent) to test the stability of self-knowledge under memory load and feedback loops.

Metrics included source attribution accuracy, reading hallucination rates (failure to reproduce external words), metacognitive sensitivity (Goodman–Kruskal $\gamma$ linking confidence to accuracy), and Signal Detection Theory (SDT) parameters ($d'$, criterion $c$, Type 2 AUC).

Key Results

• Memory Structure Dictates Performance:

  • In Experiment 1, models achieved ceiling accuracy (100%) for internal items in the Single-Turn condition because the generated word remained in the active context window, making retrieval trivial. However, Trial-Chain conditions revealed fragility: accuracy for internal items dropped significantly for some models (e.g., Gemma 3 12B fell to chance levels), while external item accuracy became variable.
  • In Experiment 2, introducing episodic delay reversed the source advantage. Unlike Experiment 1, models found external (perceived) items easier to attribute than internal (imagined) items. This suggests that when the "shortcut" of active context retrieval is removed, the diagnostic features of internal generation (richer cognitive operations) are insufficient for the models to distinguish them from external stimuli.

• The Role of Reading Hallucinations:

  • In Experiment 1, the strongest predictor of source attribution failure was the reading hallucination rate (failure to reproduce the externally provided word). If a model failed to encode the external word correctly, it could not accurately attribute the source.
  • In Experiment 2, corrective feedback paradoxically increased reading hallucination rates, particularly in larger set sizes. This indicates that feedback loops in multi-turn interactions may degrade encoding fidelity rather than improve it.

• Dissociation of Accuracy and Confidence:

  • Metacognitive Sensitivity: In Experiment 1, confidence generally tracked accuracy. In Experiment 2, under episodic delay and feedback, this link decoupled. For some models (e.g., Llama 3.3 70B), accuracy improved with feedback while metacognitive sensitivity collapsed (Type 2 AUC fell below chance). For others (e.g., Gemma 3 12B), feedback caused a complete inversion of source discrimination (systematically misattributing internal items as external and vice versa).
  • Confidence Drivers: In the episodic setting, trial-level accuracy no longer predicted confidence. Instead, confidence was driven by model architecture and context conditions, not the correctness of the specific judgment.

• Scaling and Architecture:

  • Performance did not scale linearly with aggregate parameter count. The largest model (Llama 4 16×17B) performed comparably to smaller models. The results implicate the size of individual expert networks within mixture-of-experts architectures as a more relevant variable for self-knowledge than total parameter count.

Key Contributions

1. Operationalization of RM in LLMs: The study provides the first systematic behavioral framework for testing source attribution in LLMs, directly comparable to human cognitive paradigms.
2. Identification of Memory as a Critical Variable: The paper demonstrates that "self-knowledge" in LLMs is not an invariant property but is highly sensitive to conversational memory structure (Single-Turn vs. Episodic).
3. Uncovering Hidden Failure Modes: The research reveals that standard accuracy benchmarks miss critical failures:
   • Source Reversal: Models can flip their attribution logic under memory load.
   • Confidence-Accuracy Decoupling: Models can become more accurate while simultaneously becoming less calibrated (or even anti-calibrated), rendering expressed confidence an unreliable trust signal.
   • Feedback-Induced Hallucination: Corrective feedback can degrade the fidelity of encoding external inputs, a failure mode not captured by single-turn metrics.

Significance and Claims
The authors claim that as AI systems adopt autonomous, multi-turn roles, evaluating what a model knows is insufficient; tracking where that knowledge came from is equally essential. The study suggests that current benchmarks, which often rely on minimal memory loads, overestimate real-world capacity. The findings imply that:

• Safety Implications: Source-attribution failure is a safety failure, not just a performance error. A model treating its own hallucinations as user facts cannot be trusted in agentic roles.
• Evaluation Shift: Deployment evaluations must include source-memory assessments. Minimal-load performance does not predict performance under episodic delay.
• Architecture Insights: Improving self-knowledge may require architectural changes (e.g., optimizing expert network sizes) rather than simply increasing aggregate parameter counts.
• Human-AI Parallel: The patterns observed (externalizing biases, confidence-accuracy decoupling) mirror human RM failures, suggesting that LLMs may share similar cognitive constraints regarding the attribution of internal vs. external origins.

The paper concludes that reality monitoring offers a direct, model-comparable behavioral test for these capacities, and that without such evaluation, the deployment of autonomous agents carries significant, unquantified risks regarding source confusion and hallucination.