Can LLMs Detect Their Confabulations? Estimating Reliability in Uncertainty-Aware Language Models

1. Problem Statement

Large Language Models (LLMs) are prone to confabulation: generating fluent but factually incorrect content. This risk is exacerbated in multi-turn and agentic applications (e.g., Retrieval-Augmented Generation or RAG), where model outputs are reused as context for subsequent turns.

The core problem addressed is the misalignment between model confidence and correctness. While LLMs often express high confidence (low uncertainty) when correct, they can also express high confidence when generating incorrect answers, particularly when influenced by misleading in-context information. Current uncertainty estimation methods often fail to detect these "confidently wrong" states, leading to the propagation of hallucinations in autonomous systems.

2. Methodology

The authors propose a two-part investigation: analyzing how context influences uncertainty and developing a method to detect unreliable outputs using internal model signals.

A. Context-Controlled Experimental Framework

The study evaluates model behavior under three distinct context settings using open QA benchmarks (HotpotQA and Natural Questions):

1. Without Context (WOC): The model relies solely on parametric knowledge.
2. With Correct Context (WCC): The model is provided with accurate, relevant external information.
3. With Incorrect Context (WIC): The model is provided with plausible but factually misleading information (synthesized using GPT-4.1-mini).

Uncertainty Estimation:
The authors utilize an evidential deep learning framework to compute token-level uncertainty from output logits:

• Aleatoric Uncertainty (AU): Captures data ambiguity (inherent noise).
• Epistemic Uncertainty (EU): Captures model ignorance (lack of evidence).
• These are derived by fitting a Dirichlet distribution to the top-$K$ logits of the output tokens.

B. Probing-Based Reliability Detection

To determine if LLMs can detect their own confabulations, the authors train lightweight probing classifiers on the model's internal representations.

• Input Features: Hidden states ($h_t^{(l)}$) from various layers and token positions.
• Token Selection Strategies:
  • Probe(EOS): Uses the final token's hidden state.
  • Probe(Exact): Uses tokens aligned with the ground-truth answer span.
  • Probe(EU): Selects tokens with the highest or lowest Epistemic Uncertainty.
  • Probe(AVG): Aggregates hidden states by averaging selected tokens (e.g., top-$k$ uncertain tokens).
• Goal: Predict a binary label indicating whether the generated response is factually correct.

3. Key Contributions

1. Context-Controlled Evaluation Framework: A systematic setup to isolate the effects of context quality on model behavior, revealing how models transition between correct and incorrect regimes based on input information.
2. Discovery of Confidence-Correctness Misalignment: Empirical evidence showing that misleading context induces overconfident confabulations. Models often produce incorrect answers with lower epistemic uncertainty (higher confidence) when exposed to misleading context compared to their baseline behavior without context.
3. Uncertainty-Guided Probing Method: A novel approach for reliability detection that leverages internal activations. The study demonstrates that aggregating hidden states from high-uncertainty tokens yields significantly better detection performance than direct uncertainty metrics or standard baselines.

4. Key Results

Impact of Context on Behavior

• Correct Context (WCC): Significantly improves answer accuracy and reduces epistemic uncertainty (increases confidence).
• Misleading Context (WIC): Causes a dramatic shift. Models transition from correct answers (in WOC) to consistently wrong answers with high confidence (low uncertainty).
  • Observation: In the WIC setting, the distribution of Epistemic Uncertainty shifts leftward (indicating higher confidence) and becomes narrower, even though the answers are wrong. This confirms that models cannot internally distinguish between valid and invalid external context.

Reliability Detection Performance

The authors evaluated various methods across three models (Fanar1-9b, Qwen2.5-7B, Gemma3-12B) and three datasets (TruthfulQA, TriviaQA, Math).

• Superiority of Probing: Probing-based methods consistently outperformed direct uncertainty baselines (e.g., LogProb, $P(True)$, and LogTokU).
• Best Strategy: Probe(AVG), which aggregates hidden states from tokens selected based on uncertainty (specifically the top-5 uncertain tokens), achieved the highest AUROC scores.
  • Example: On the TriviaQA dataset, Probe(AVG) achieved an AUROC of 0.765 (Fanar), 0.786 (Qwen), and 0.818 (Gemma), significantly outperforming LogTokU and $P(True)$.
• Layer Sensitivity: Aggregated features from middle-to-upper layers provided the most robust signals for reliability detection.

5. Significance and Implications

• Limitations of Direct Uncertainty: The study highlights that raw uncertainty signals (logits) are insufficient for detecting hallucinations in RAG or agentic settings. A model can be "confidently wrong" if the context is misleading.
• Internal State Utility: The model's internal hidden states contain richer information about reliability than the final output probabilities. By focusing on uncertainty-guided token selection, one can construct a "reliability detector" that identifies when a model is likely hallucinating, even if the model itself expresses high confidence.
• Future Directions: The findings suggest that future safety mechanisms for LLMs should not rely solely on output calibration but must incorporate context validation and internal probing to prevent the propagation of confabulated content in multi-turn interactions.

In summary, while LLMs cannot inherently "know" they are hallucinating when fed misleading context, their internal representations (specifically aggregated hidden states of uncertain tokens) can be effectively probed to detect these failures, offering a pathway toward more reliable, uncertainty-aware generation systems.