💬 NLP

Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models

This paper introduces SinkProbe, a state-of-the-art hallucination detection method that leverages attention sinks—tokens accumulating disproportionate attention mass—as internal signals indicating a shift from input-grounded to prior-dominated computation, while also revealing that effective detection relies on sinks with large value vector norms and mathematically unifying previous approaches with this mechanism.

Original authors: Jakub Binkowski, Kamil Adamczewski, Tomasz Kajdanowicz

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jakub Binkowski, Kamil Adamczewski, Tomasz Kajdanowicz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are reading a story written by a very confident, incredibly fast robot. This robot has read almost every book ever written and can answer questions about anything from ancient history to quantum physics. But sometimes, when the robot gets stuck or doesn't know the answer, it doesn't say, "I don't know." Instead, it keeps talking in a smooth, fluent voice, making up facts that sound perfect but are completely made up. In the world of computer science, this is called a "hallucination." It's a major problem because if we trust these robots for important tasks like medical advice or legal decisions, their confident lies could cause real trouble.

To understand how to catch these lies, we first need to peek inside the robot's brain. Modern robots like this use a system called a "Transformer," which works by paying attention to different parts of a sentence to figure out what comes next. Think of it like a group of detectives (called "attention heads") looking at a crime scene (the sentence). Usually, they focus on the most important clues. However, researchers recently discovered something weird: sometimes, these detectives get obsessed with a few specific, boring clues—like the very first word of the sentence or a blank space—ignoring the rest of the evidence. They call these obsessions "attention sinks." It's as if the detectives are staring so hard at a single dust bunny that they miss the actual murder weapon.

This paper, titled "Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models," asks a simple but brilliant question: Can we use these weird "attention sinks" to catch the robot lying? The authors, Jakub Binkowski and his team, propose a new method called SinkProbe. They suggest that when a robot starts to hallucinate, its internal attention system gets "clogged" or "collapsed" onto these unimportant tokens, losing its connection to the real facts. By measuring how much attention the robot is dumping onto these "sinks," they can predict if the robot is about to start making things up.

The Detective's New Tool: SinkProbe

The authors didn't just guess; they built a tool to test this idea. They looked at the attention maps of several popular large language models (LLMs) like Llama and Mistral while they were answering questions on various tricky datasets, including math problems and general knowledge quizzes.

Here is how their method works, using a simple analogy: Imagine the robot's brain is a busy highway with many lanes (layers) and many cars (tokens). Usually, the cars spread out, looking at different parts of the road. But when the robot is about to hallucinate, the traffic jams up. A few specific cars (the "sinks") get stuck in a massive traffic jam, and almost every other car on the highway starts staring at them instead of the road ahead.

The authors developed a score called the Sink Score to measure exactly how much traffic is jamming up at these specific spots. They found that when a robot is lying, these sink scores spike. But there's a twist: not all traffic jams are equal. The authors discovered that the "sinks" that really matter for catching lies are the ones that are not just getting a lot of attention, but are also carrying a heavy load (a large "value vector"). It's like finding a traffic jam where the stuck cars are also carrying heavy crates; that's when you know something is seriously wrong with the flow of information.

What They Found (and What They Didn't)

The results were quite promising. When they trained a simple computer program (a logistic regression classifier) to look at these sink scores, it became very good at spotting hallucinations. In fact, on most of the tests they ran, their new method, SinkProbe, performed better than other existing methods that tried to detect lies by looking at attention patterns in different ways.

For example, on a dataset called GSM8K (which has math problems), SinkProbe achieved a score of 0.845, beating the previous best method which scored 0.835. On TruthfulQA, a dataset designed to trick models into lying, SinkProbe scored 0.785, again coming out on top. The authors noted that this method works well across different sizes of models, from smaller 3-billion-parameter models up to larger 12-billion-parameter ones.

However, the paper is careful not to claim that they have "solved" the problem of hallucinations. They explicitly state that their method requires access to the model's internal attention weights, which means it only works on open-source models where you can see the brain's wiring, not on closed, secret models. They also clarify that while they found a strong connection between attention sinks and hallucinations, they haven't proven that the sinks cause the lies. It's like seeing that a car's engine is overheating when it crashes; the heat is a clear signal of trouble, but the heat itself might not be the reason the car crashed.

Why This Matters

The beauty of this work is that it unifies several different ideas. The authors showed that other methods people have tried—like looking at the "shape" of the attention graph or comparing how much attention is paid to the question versus the answer—are actually just different ways of looking at the same "sink" phenomenon. It's like realizing that three different people are all trying to describe a elephant by touching its leg, its ear, and its tail; they are all feeling the same animal, just from different angles.

By focusing on these "attention sinks," the authors provide a simpler, more direct way to listen to the robot's internal thoughts. They found that the robot's "lies" are often signaled by a specific kind of internal traffic jam where the brain stops processing new information and starts recycling old, unimportant signals. While this doesn't fix the robot's tendency to lie, it gives us a very effective alarm system to know when the robot is starting to drift off into fantasy.

In short, the paper suggests that if you want to know if a super-smart AI is telling the truth, you shouldn't just listen to what it says. You should also watch where it's looking. If it's staring too hard at the wrong things, it's probably making things up.

Technical Summary: Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models

Problem Statement
Large Language Models (LLMs) frequently produce hallucinations: fluent, confident outputs that are factually incorrect or unsupported by the input context. While recent detection methods utilize features derived from attention maps, hidden representations, or spectral properties, the underlying mechanisms they exploit remain poorly understood. Existing approaches often require external knowledge sources, multiple generations, or complex computational overhead, providing limited insight into the internal computational failures that give rise to hallucinations. Specifically, the relationship between "attention sinks"—tokens that accumulate disproportionate attention mass—and hallucination remains under-explored as a primary diagnostic signal.

Methodology: SinkProbe
The authors propose SinkProbe, a hallucination detection method grounded in the observation that hallucinations are entangled with attention sinks. The method operates purely on internal attention weights without requiring external corpora or multiple sampling.

Sink Score Definition: The core metric is the sink score, defined for a specific token $i$ in layer $l$ and head $h$ as the average attention weight received from all subsequent tokens $u \ge i$ , normalized by the remaining sequence length. High sink scores indicate tokens that persistently attract attention as generation progresses.
Feature Construction: Rather than relying on token identity, the method aggregates sink scores using order statistics. For each attention head, the top- $k$ sink scores are sorted and retained. These values are concatenated across all layers and heads to form a fixed-dimensional feature vector $z$ .
Classification: A lightweight logistic regression classifier (the "probe") is trained on these features to distinguish between hallucinated and factually correct outputs.
Role of Value Vectors: The authors investigate the computational impact of sinks, finding that not all high-attention tokens are equally relevant. The hallucination signal is strongest when high attention concentration coincides with unusually large value vector norms. These "computationally active sinks" dominate the attention output and induce compressed, prior-dominated representations.
Theoretical Unification: The paper establishes a mathematical relationship showing that several existing attention-based detectors (e.g., LLMCheck, LookbackLens, Topological Divergence, and Spectral features) implicitly depend on attention sink behavior. For instance, spectral features (eigenvalues of the Laplacian) are shown to correspond to sink scores discounted by self-attention.

Key Contributions

SinkProbe: A novel, interpretable method for hallucination detection derived solely from attention maps.
Identification of Computationally Active Sinks: The work demonstrates that sink-based signals are most predictive when attention concentration aligns with large value vector norms, suggesting hallucinations stem from breakdowns in internal information flow rather than mere knowledge gaps.
Unification of Prior Work: The authors provide a theoretical framework unifying various spectral and graph-based detection methods, showing they are transformations of sink behavior.
State-of-the-Art Performance: The method achieves superior or competitive results across diverse datasets and model families compared to existing baselines.

Experimental Results
The method was evaluated on seven benchmarks (GSM8K, UMWP, TruthfulQA, TriviaQA, NQ-Open, SQuADv2, HaluEvalQA) covering question answering and mathematical reasoning. Four LLMs were tested: Llama3.2-3B, Phi3.5, Llama3.1-8B, and Mistral-Nemo (12B).

Performance: SinkProbe achieved the best ROC-AUC score in 23 out of 28 model-dataset pairs. While it delivered stronger average performance and was best in the vast majority of settings, it did not outperform all baselines in every instance; for example, LookbackLens achieved higher scores on GSM8K and SQuADv2 when using the Phi3.5 model. It generally outperformed baselines such as AttentionScore, LookbackLens, MTopDiv, and spectral methods (AttnEigval, LapEigval).
Feature Sparsity: Analysis of the logistic regression coefficients revealed that the probe relies on a very small subset of features (1% to 4% of all sink scores), primarily concentrated in the middle and final layers of the network.
Top-k Sensitivity: The method achieves peak performance with small values of $k$ (typically $k < 10$ ), indicating that hallucination signals are highly concentrated in a few dominant sinks.
Token Semantics: Semantic analysis of top-ranked sinks shows they are predominantly special tokens (e.g., <bos>), whitespace, or punctuation, rather than meaningful content words, reinforcing the idea that hallucinations are linked to structural attention collapse rather than specific semantic errors.

Significance and Claims
The paper claims that describing information flow in LLMs through the lens of sink scores offers a more fundamental perspective on hallucination detection. By identifying attention sinks as a key internal correlate of hallucination, the authors argue that several existing methods can be understood as implicitly leveraging this concept.

The authors position SinkProbe as a simple, direct, and unifying signal that explains the effectiveness of prior attention-based approaches. They emphasize that while the method provides a strong correlational signal, it does not yet establish causality; future work is needed to verify if manipulating sink formation can causally reduce hallucinations. The method is currently limited to open-weight models where attention weights are accessible, though the authors suggest efficient deployment could be achieved via specialized kernels.

Ultimately, the work contributes a lightweight, interpretable detector that supports safer LLM deployment by enabling monitoring and selective abstention without relying on external retrieval or repeated sampling, while simultaneously advancing mechanistic research into how attention routing relates to model failure modes.

The Detective's New Tool: SinkProbe

What They Found (and What They Didn't)

Why This Matters

Technical Summary: Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models

More like this