💬 NLP

Detecting Hallucinations in Large Language Models via Internal Attention Divergence Signals

This paper proposes a lightweight, single-pass method for detecting LLM hallucinations by using Kullback-Leibler divergence in attention matrices as a predictive, interpretable signal of uncertainty, which performs competitively with existing methods without requiring repeated sampling or external models.

Original authors: Gijs van Dijk

Published 2026-05-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gijs van Dijk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Confident Liar"

Imagine you are asking a very smart, well-read friend for advice. Sometimes, they give you the perfect answer. Other times, they confidently make up a story that sounds great but is completely false. This is what we call a hallucination in AI.

The tricky part is that the AI doesn't know it's lying. It sounds just as confident when it's wrong as when it's right. Usually, to catch a lie, you'd have to ask the AI the same question ten times and see if it gives different answers (like asking a witness to recount a story multiple times). But that takes a lot of time and computer power.

The Solution: Listening to the "Internal Monologue"

This paper proposes a new, super-fast way to catch these lies. Instead of waiting for the AI to finish its answer and then checking it, the authors look at the AI's internal "brain waves" while it is thinking.

In AI models, there is a mechanism called Attention. You can think of Attention like a spotlight. When the AI is answering a question, this spotlight shines on different words in its memory to decide which ones are important for the next word it will write.

When the AI knows the answer: The spotlight is focused and steady. It shines brightly on the specific facts it needs (like a person's name or a date).
When the AI is guessing or lying: The spotlight gets shaky, scattered, or wanders aimlessly. The AI doesn't know where to look, so it spreads its attention everywhere.

The "Spotlight Test" (The Method)

The authors created a simple math test to measure how "shaky" this spotlight is.

The Uniform Baseline: Imagine a room with 100 people. If you are completely unsure who to talk to, you might look at everyone equally. This is a "uniform" distribution (1/100th of your attention on each person). This represents maximum uncertainty.
The Measurement: The authors measure the difference between the AI's actual spotlight and this "scattered" baseline. They call this KL Divergence.
- High Divergence: The spotlight is very focused (shining hard on a few specific words). This usually means the AI is confident and likely correct.
- Low Divergence: The spotlight is scattered (looking at everyone equally). This means the AI is confused or guessing.

Wait, isn't the paper saying the opposite?
Actually, the paper found a nuance: When the AI is hallucinating, it often tries to force a confident answer on a topic it doesn't know. In these cases, the attention patterns get weirdly concentrated on the wrong things, or the math shows a specific "divergence" signal that flags the error. Essentially, the math detects that the AI's internal "focus" doesn't match the pattern of a truthful, well-grounded answer.

The "Lightweight Detective"

The authors didn't just look at the spotlight; they built a tiny, simple detector (a "logistic regression probe") to read these signals.

No Re-asking: This method only needs the AI to answer once.
No Extra Models: It doesn't need a second AI to check the first one.
Fast: It happens in a single pass, like reading a sentence once and instantly knowing if it feels "off."

What They Found

The researchers tested this on different types of questions (facts, math, reasoning) and different AI models. Here is what they discovered:

The "Middle Brain" is Key: The signals that tell us if an answer is true or false are mostly found in the middle layers of the AI's brain. The beginning and end layers are less helpful for this specific task.
Facts are the Trigger: The "shaky spotlight" signal is strongest when the AI is talking about facts—specifically names, dates, and numbers. If the AI is just chatting about "the" or "and" (stop words), the signal is quiet. But when it tries to name a person or a year, the signal screams if it's unsure.
It's a Team Effort: The signal isn't coming from just one single "lie detector" neuron. It's a chorus of many different parts of the AI working together. If you remove one part, the system still works, but if you remove the whole middle section, the detector goes blind.

The Bottom Line

This paper shows that we can catch AI hallucinations by listening to its internal "spotlight" rather than just judging its final words. It's like checking if a student is actually thinking about the answer or just guessing by looking at where their eyes are darting, rather than waiting to see if they get the grade right.

This method is fast, cheap, and works across many different types of questions, offering a "white-box" window into the AI's confidence without needing to run it multiple times.

Technical Summary: Detecting Hallucinations in Large Language Models via Internal Attention Divergence Signals

1. Problem Statement

Large Language Models (LLMs) frequently generate fluent but factually incorrect content, a phenomenon known as hallucination. This poses significant risks in high-stakes domains such as healthcare and law. A primary challenge in addressing this issue is that autoregressive LLMs are trained to generate fluent continuations rather than calibrated confidence estimates. Consequently, standard output-based uncertainty signals (e.g., perplexity, log-likelihood, or token probabilities) often fail to distinguish between confident errors and reliable knowledge, as hallucinated statements can receive high likelihoods under the model distribution.

Existing uncertainty quantification methods often rely on repeated sampling (e.g., semantic entropy) or external verifier models, which introduce substantial computational overhead. Furthermore, approaches relying on hidden state probing often struggle to generalize across tasks, capturing task-specific correlations rather than global confidence signals. There is a need for a lightweight, single-pass method that extracts uncertainty signals directly from the model's internal mechanisms without requiring external models or multiple generations.

2. Methodology

The proposed method introduces a single-pass uncertainty quantification technique based on attention divergence. The core hypothesis is that hallucinations arise from failures in internal computation, specifically characterized by misallocated or diffuse attention patterns when the model lacks reliable knowledge.

2.1 Attention Divergence Signal

The method utilizes the Kullback–Leibler (KL) divergence to measure the deviation of an attention head's distribution from a uniform reference distribution.

Reference Distribution ( $U$ ): A uniform distribution over the context tokens, representing a state of maximal uncertainty where the model attends equally to all positions.
Attention Distribution ( $P$ ): The discrete probability distribution over token positions generated by a specific attention head at a given generation step.
Metric: The KL divergence is calculated as $D_{KL}(P \parallel U) = \log T - H(P)$ , where $T$ is the number of tokens in the context window and $H(P)$ is the entropy of the attention distribution.

Intuitively, a high concentration of attention (low entropy) on informative tokens results in high KL divergence, suggesting confidence. Conversely, diffuse attention (high entropy) results in low divergence, suggesting uncertainty. The authors posit that hallucinations may manifest as wrongly calibrated confidence (high divergence on misleading tokens) or, in the context of the probe's aggregation, that the aggregate signal of divergence patterns correlates with correctness.

2.2 Feature Extraction and Probing

Feature Construction: For each generated answer, the mean KL divergence is computed for every attention head across all generation steps. These values form a feature vector $x_i \in \mathbb{R}^{L \times H}$ , where $L$ is the number of layers and $H$ is the number of heads.
Probe Training: A lightweight logistic regression probe with Lasso (L1) regularization is trained to predict answer correctness ( $y=1$ for correct, $y=0$ for incorrect) based on these divergence features. The L1 regularization serves to select a sparse subset of the most predictive attention heads, providing interpretability regarding which parts of the model encode the signal.
Evaluation: Performance is measured using AUROC (Area Under the Receiver Operating Characteristic curve), Expected Calibration Error (ECE), and accuracy.

3. Key Contributions

Single-Pass Uncertainty Estimation: The method operates in a single forward pass, extracting uncertainty signals directly from attention matrices without requiring repeated sampling or external models.
Interpretability: By analyzing the KL divergence of attention distributions, the method provides a "white-box" signal. The authors demonstrate that the signal is not random but concentrated in specific architectural components.
Generalizability: The approach is validated across multiple model families (Llama-3.2, Qwen3, Mistral) and diverse task types (factual QA, multi-hop reasoning, mathematical reasoning).

4. Experimental Results

4.1 Performance Across Datasets

The method was evaluated on four datasets: TruthfulQA (multiple-choice factual QA), TriviaQA (open-domain factual QA), HotpotQA (multi-hop reasoning), and GSM8K (mathematical reasoning).

Factual QA: The method achieved strong performance on TruthfulQA (AUROC > 0.89 for all models) and TriviaQA (AUROC > 0.83).
Reasoning Tasks: Performance on HotpotQA ranged from 0.77 to 0.80 AUROC. On GSM8K, results varied by model, with Qwen3-4B achieving 0.945 AUROC, while Llama and Mistral showed moderate but significant results above chance.
Comparison to Baselines: On HotpotQA, the method (AUROC 0.78) outperformed sampling-based methods (e.g., Semantic Entropy, SelfCheckGPT) and other single-pass baselines (e.g., Perplexity, TOHA). It also showed competitive or superior performance compared to hidden state probing methods on TruthfulQA and TriviaQA.

4.2 Ablation and Analysis

Layer and Head Localization: The signal is not uniformly distributed. Ablation studies revealed that removing middle layers caused the most significant performance drop, indicating the signal is concentrated there. Furthermore, the signal is distributed across multiple correlated heads rather than being localized to a single "hallucination head."
Token Specificity: The divergence signal is most pronounced on factual tokens (named entities and numbers). Stop words and punctuation showed low divergence. Analysis of the extreme tail (99th percentile) showed that ~73% of high-divergence words were named entities.
Prompt Influence: A portion of the uncertainty signal is present even before generation begins (computed solely over prompt tokens), suggesting that some uncertainty is aleatoric (related to the prompt's ambiguity) rather than solely epistemic (model knowledge gaps).
Sanity Checks: Permuting correctness labels and testing against simple output properties (length, punctuation) confirmed that the signal is not an artifact of data leakage or superficial output characteristics.

5. Significance and Claims

The paper claims that internal attention dynamics provide an efficient and interpretable signal for detecting model uncertainty and hallucinations. The primary significance lies in the demonstration that:

Hallucinations are detectable via internal states: The failure to retrieve reliable knowledge manifests as specific patterns in attention divergence, distinct from the fluent but incorrect outputs produced by the model.
Efficiency: The method offers a computationally cheap alternative to sampling-based uncertainty estimation, making it viable for real-time applications where latency is critical.
Architectural Insight: The findings suggest that uncertainty is encoded in the middle layers and on factual anchors, supporting the view that hallucinations are structural failures in internal computation rather than mere decoding errors.

6. Limitations

The authors explicitly note several limitations:

Correlation vs. Causation: The study establishes a predictive correlation between attention divergence and correctness but does not prove a causal relationship.
Probe Dependency: The method relies on a logistic regression probe to aggregate signals. The specific subset of heads selected varies across datasets and model families, suggesting the probe may be too specific and that a universal, probe-free rule for uncertainty has not yet been identified.
White-Box Requirement: The method requires access to internal attention weights, rendering it inapplicable to black-box models that only provide generated text or output probabilities.
Interpretability: While the probe identifies informative heads, the resulting model remains difficult to interpret as a complete explanation of how uncertainty is represented, serving more as a signal extraction tool than a full mechanistic explanation.