💻 computer science

When Context Bites: Detecting RAG Poisoning via Document-Level Attention Collapse

This paper introduces D-SCAN, a lightweight detection framework that identifies RAG poisoning attacks by monitoring document-level attention collapse, a distinctive signature where attention entropy decreases as the model focuses on adversarial documents, overcoming the limitations of existing output-side detection methods.

Original authors: Yingtao Ren, Ziyi Zhao, Yiwei Fu, Xiao Luo, Yu-Cheng Chang, Chin-Teng Lin

Published 2026-08-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yingtao Ren, Ziyi Zhao, Yiwei Fu, Xiao Luo, Yu-Cheng Chang, Chin-Teng Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern landscape of artificial intelligence, large language models act as vast libraries of human knowledge, capable of answering questions and generating text with startling fluency. However, these models have a blind spot: they do not inherently know the latest facts or specific details about the world unless they are told. To fix this, developers use a system called retrieval-augmented generation. Think of this process as a student taking an open-book test; the computer first searches a database for relevant documents, feeds them to the model, and then asks the model to write an answer based on that new information. This method allows the machine to access up-to-date facts, but it also opens a door for trouble. If an attacker slips a carefully crafted, false document into that database, the model might read it, believe it, and use it to construct a harmful or incorrect answer. This is known as a poisoning attack, and it poses a serious risk in high-stakes fields like law and medicine, where a single wrong fact could have real-world consequences.

For a long time, researchers trying to catch these attacks looked at the final answer the computer produced. They hoped to spot the lie by checking if the answer sounded unsure or if it contradicted itself. The prevailing idea was that when a model is confused or hallucinating, it would sound hesitant, like a person guessing. However, a team of researchers from the University of Technology Sydney and Peking University discovered that this assumption is dangerously wrong. They found that when a model is successfully tricked by a poisoned document, it does not sound confused at all. Instead, it sounds more confident than usual. The attackers design their fake documents so perfectly that the model becomes "blindly confident," assigning a higher probability to the wrong answer than it ever would to a correct one. This means that traditional methods, which rely on detecting uncertainty or inconsistency, are often useless because the model is not uncertain; it is convinced it is right.

To solve this, the researchers stopped looking at the final text and started looking inside the machine's brain while it was thinking. They examined how the model paid attention to the different documents it was reading. In a normal, healthy reading session, a model spreads its attention across several documents, weighing the evidence from each one to build a complete picture. The researchers found that when a poisoning attack occurs, this behavior changes drastically. The model stops looking at the other documents and focuses almost entirely on the single poisoned file. They call this phenomenon "attention collapse." It is as if the model's gaze is hijacked, locking onto the malicious information and ignoring everything else. This happens even if the attack fails to change the final answer, making it a reliable early warning sign that something is wrong.

Based on this discovery, the team built a new detection tool called D-SCAN. This system is designed to be lightweight and fast, monitoring the internal attention patterns of the model as it works. Instead of waiting to see if the final answer is wrong, D-SCAN watches for that specific moment when the model's focus collapses onto a single document. In their tests, the researchers used three different sets of complex questions and found that their new method was far better at spotting attacks than any existing technology. While other tools struggled to tell the difference between a real answer and a fake one, D-SCAN consistently identified the poisoning attempts. Perhaps most importantly, the tool worked even when the attack did not succeed in changing the final output. This suggests that the act of the model being hijacked is a distinct signature that exists independently of the final result.

The study also revealed a surprising detail about the size of the models themselves. One might assume that a larger, more powerful computer would be better at spotting a trick, but the researchers found the opposite. Smaller models were actually better at detecting the poison, while the largest models seemed more easily fooled. This suggests that the very training that makes these large models follow instructions so well might also make them too trusting of the documents they are given. By focusing on the internal mechanics of how attention is distributed rather than the surface-level text, the researchers have provided a new way to secure these systems. Their work shows that to protect artificial intelligence, we must understand not just what it says, but how it thinks.

Technical Summary: When Context Bites: Detecting RAG Poisoning via Document-Level Attention Collapse

Problem Statement

Retrieval-Augmented Generation (RAG) systems, which enhance Large Language Models (LLMs) with external knowledge, are increasingly vulnerable to poisoning attacks. In these scenarios, adversaries inject malicious documents into the retrieval database to manipulate the generator into producing specific, harmful outputs.

Current detection methods primarily rely on post-hoc inspection of the final generated text or implicit output-side signals, such as perplexity and consistency metrics. However, this paper identifies a critical flaw in these approaches: adversarial contexts are often optimized to maximize generation likelihood, inducing a state of "false confidence." Consequently, poisoned outputs frequently exhibit lower perplexity and higher token probabilities than benign ones, rendering uncertainty-based detection ineffective. Furthermore, existing methods lack interpretability and fail to account for the granular nature of RAG contexts, where information is sparsely distributed across discrete retrieved chunks.

Methodology: D-SCAN

To address these limitations, the authors propose D-SCAN (Document-level Signal Collapse Analysis), a lightweight, interpretable detection framework that shifts the paradigm from examining output semantics to inspecting the internal dynamic signals of the LLM generation process.

Key Mechanisms

Blind Confidence Phenomenon: Empirical analysis reveals that poisoning attacks do not confuse the model; rather, they induce "blind confidence." Poisoned samples demonstrate higher mean token probabilities and lower variance compared to clean samples, causing the model to stochastically collapse into the attacker's target output with high certainty.
Attention Collapse: The core discovery is a distinctive neural signature termed Attention Collapse.
- In benign generations, the model allocates attention relatively evenly across multiple relevant retrieved documents.
- In attacked generations, the model's attention is disproportionately hijacked by the poisoned documents, suppressing factual evidence.
- This manifests as a sharp reduction in document-level attention entropy.
Document Attention Density: To quantify this, the authors introduce a metric called Document Attention Density. This aggregates token-level attention weights for a specific document and normalizes them by the document's length.
Detection Framework: D-SCAN extracts features directly from the LLM's internal states during inference, specifically utilizing attention distribution metrics (entropy, variance, and density) at both the token and document levels. These features are fed into a lightweight linear classifier to identify attacks.

Key Contributions

The paper outlines three primary contributions:

Mechanistic Exploration: The identification of Attention Collapse as a reliable neural signature of poisoning attacks. The authors demonstrate that adversarial intent is exposed within the attention mechanism itself, even when the final output appears benign.
Methodological Innovation: The proposal of D-SCAN, a novel framework that formalizes the shift from output-level checking to internal dynamic signal inspection.
Robust Performance: The demonstration that D-SCAN can detect adversarial intent regardless of whether the attack successfully alters the final answer, maintaining detection fidelity even in "attack failure" scenarios.

Experimental Results

The authors evaluated D-SCAN on three multi-hop QA benchmarks: HotpotQA, 2Wiki, and Musique, using the Llama-3.1-8B-Instruct model and PoisonedRAG for attack generation.

Superior Accuracy: D-SCAN consistently outperformed state-of-the-art baselines, including specialized detectors (HaloScope, ReDeep, RevPRAG) and zero-shot LLM detectors. For instance, on the 2Wiki benchmark, D-SCAN achieved an AUC of 0.9337 and an F1 score of 0.8578, significantly surpassing the next best method (ReDeep at 0.8403 AUC).
Inverse Scaling in Zero-Shot Detection: The study observed that larger LLMs (e.g., Qwen2.5-14B, Qwen3-30B) did not necessarily perform better as zero-shot detectors; in some cases, smaller models (Qwen2.5-7B) outperformed larger counterparts, suggesting that extensive instruction alignment may inadvertently increase trust in retrieved context.
Robustness to Attack Failure: Crucially, D-SCAN maintained high detection performance even when attacks failed to produce the target harmful answer. This confirms that attention collapse is an intrinsic signature of the poisoned document, independent of the behavioral outcome.
Efficiency: The method is effective with a single generation sample ( $N=1$ ), achieving AUC scores above 0.8, making it suitable for real-time inference without the need for computationally expensive multi-sampling.
Ablation Studies: Removing document-level attention metrics caused a more significant performance drop than removing token-level metrics, confirming that cross-document attention hijacking is the dominant signal.

Significance and Claims

The paper claims that D-SCAN represents a fundamental shift in RAG security by enabling mechanism-level detection rather than relying on output-side checks. By capturing specific attention patterns, the system can identify poisoning attempts before they manifest as observable errors. The authors emphasize that their approach is lightweight, interpretable (allowing for the localization of the specific poisoning document), and robust against attacks that successfully induce false confidence. The work aims to inspire further research into mechanism-aware defenses for trustworthy RAG systems.