🤖 AI

Addressing Corpus Knowledge Poisoning Attacks on RAG Using Sparse Attention

This paper introduces Sparse Document Attention RAG (SDAG), a novel defense mechanism that employs block-sparse attention to prevent harmful cross-document interactions in Retrieval Augmented Generation systems, thereby significantly mitigating corpus knowledge poisoning attacks and outperforming existing state-of-the-art defenses.

Original authors: Sagie Dekel, Moshe Tennenholtz, Oren Kurland

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sagie Dekel, Moshe Tennenholtz, Oren Kurland

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern landscape of artificial intelligence, large language models act as vast repositories of human knowledge, capable of answering questions and generating text with startling fluency. However, these models have a limitation: they are trained on data that stops at a certain point in time, meaning they cannot know about recent events or specific, niche facts without help. To solve this, researchers developed a system called Retrieval-Augmented Generation. In this setup, when a user asks a question, the system first searches a massive library of documents to find relevant information. It then feeds both the question and these retrieved documents into the language model, which uses them as a guide to craft an accurate answer. This method keeps the AI's knowledge fresh and reduces the chance of it making things up, a problem known as hallucination. Yet, this very reliance on external documents creates a new vulnerability. Because the system trusts the documents it finds, a malicious actor could secretly inject false information into the library. If the system retrieves these poisoned documents, it might be tricked into ignoring the truth and delivering a lie instead.

Researchers at the Technion in Israel have identified a specific weakness in how these systems process information that makes them susceptible to such deception. In standard language models, the mechanism that allows the AI to focus on different parts of a text is designed to let every word look back at every previous word in a continuous stream. This works well for writing a story or summarizing a single article, where context flows naturally from one sentence to the next. However, in a retrieval system, the input is often a collection of separate, distinct documents pasted together. The researchers argue that the standard method of processing these documents allows words from one document to interact with words from another in a way that can be harmful. When a malicious document is present, its misleading content can influence the AI's understanding of the truthful documents, effectively poisoning the entire response.

To address this, the team introduced a new defense strategy called Sparse Document Attention. Instead of allowing the AI to let every word in the retrieved set look at every other word, this method creates a strict boundary between the documents. It forces the system to treat each retrieved document as an isolated island. Within a single document, the words can still reference one another to maintain context, but they are completely blocked from looking at or being influenced by words in any of the other retrieved documents. This change is applied only when the system is generating an answer, requiring no retraining of the underlying AI model and no complex additions to the software pipeline. It is a simple adjustment to the rules of attention that prevents the harmful cross-talk between a liar and a truth-teller.

The researchers tested this approach rigorously across a variety of scenarios, using different question-answering datasets and multiple types of language models. They simulated attacks where an adversary injected misleading documents designed to steer the AI toward a specific, incorrect answer. In these tests, the standard system, which allowed documents to influence one another, frequently fell for the trick, often producing the attacker's desired false answer. In contrast, the system using the new sparse attention method proved far more resilient. It successfully ignored the poisoned documents in the vast majority of cases, maintaining the accuracy of its answers and drastically reducing the success rate of the attacks. The improvement was so significant that the new method outperformed existing, more complex defense strategies that had been developed to catch these attacks.

The study also explored how the position of a fake document affects the outcome. They found that when a deceptive document is very similar in content to the truthful ones, it is harder for the system to resist its influence. However, the new method remained effective even in these difficult situations. Furthermore, the researchers discovered that this defense works well even when the AI is asked to solve complex problems that require combining information from multiple documents, such as multi-step reasoning questions. While the initial setup showed a slight drop in performance compared to the standard method, the researchers found that a brief period of fine-tuning the model to the new rules completely recovered this loss, resulting in a system that was both robust against attacks and highly accurate.

The findings suggest that the way an AI looks at its source material is just as important as the material itself. By simply preventing different documents from talking to each other, the system becomes much harder to fool. This approach offers a practical and efficient way to secure retrieval-based AI systems, ensuring that they rely on the truth rather than being swayed by the most persuasive lie in the room. The work establishes a new standard for protecting these systems, showing that a fundamental change in how the model processes information can be more effective than adding layers of complex detection tools.

Technical Summary: Addressing Corpus Knowledge Poisoning Attacks on RAG Using Sparse Attention

Problem Statement

Retrieval-Augmented Generation (RAG) is a dominant paradigm for updating Large Language Model (LLM) knowledge and reducing hallucinations. However, RAG systems are vulnerable to corpus knowledge poisoning attacks, where an adversary injects misleading or malicious documents into the retrieval corpus to steer the LLM's output toward a desired, incorrect response.

The authors argue that a fundamental, often overlooked vulnerability in RAG stems from the standard causal attention mechanism used in decoder-only LLMs. In a standard RAG pipeline, retrieved documents are concatenated into a single prompt. Under causal attention, tokens in one retrieved document can attend to (and be influenced by) tokens in other retrieved documents. When the retrieved set contains conflicting information—such as correct benign documents versus injected adversarial documents—this cross-document attention allows the adversarial content to corrupt the hidden states of benign documents, leading to incorrect generation.

Methodology: Sparse Document Attention RAG (SDAG)

To mitigate this vulnerability, the authors propose Sparse Document Attention RAG (SDAG), a defense mechanism that modifies the attention mask during inference without requiring model fine-tuning or architectural changes.

Core Mechanism

SDAG enforces a block-sparse attention pattern over retrieved documents:

Intra-document Attention: Tokens within a single retrieved document can attend to previous tokens within the same document.
Inter-document Blocking: Cross-attention between different retrieved documents is explicitly disabled. Tokens in document $d_i$ cannot attend to tokens in document $d_j$ (where $i \neq j$ ).
Preserved Context: Tokens in the task instruction block ( $B_T$ ) and the generator's context block ( $B_C$ ) retain standard causal attention, allowing them to attend to all preceding tokens. Similarly, generated tokens can attend to all previous tokens in the sequence.

Implementation

Minimal Overhead: The method requires only a modification to the attention mask at inference time.
Compatibility: It is compatible with any decoder-only LLM and can be integrated with existing pre-generation defense methods (e.g., document filtering or discrimination).
No Training: Unlike many defense strategies, SDAG does not require fine-tuning the generator or training additional discriminators.

Experimental Setup

The authors conducted a rigorous empirical evaluation across multiple dimensions:

Datasets: HotpotQA (multi-hop), TriviaQA, and Natural Questions (NQ) (single-hop), using the English Wikipedia dump as the corpus.
Models: Three generator LLMs (Llama-8B, Qwen-7B, Mistral-7B) and various retrievers (E5, Contriever, BM25).
Attack Strategies:
- In-Set vs. In-Corpus: Evaluating scenarios where adversarial documents are guaranteed to be in the retrieved set versus where they are injected into the corpus and may or may not be retrieved.
- Document Count: Single-document attacks and multiple-document attacks (2 or 3 adversarial documents).
- Spatial Strategies: Random selection, "Near" (adversarial documents close to benign documents in embedding space), and "Far" (distant in embedding space).
- Adaptive Attacks: Scenarios where the attacker observes benign documents or previous model responses to refine the attack.
Baselines: Standard Causal Attention RAG (CARG), RAGDefender (filtering-based), and Discern&Answer (discriminator-based).
Metrics: Accuracy (ACC) and Attack Success Rate (ASR).

Key Results

1. Superiority Over Standard Attention (RQ1)

SDAG consistently outperforms the standard Causal Attention RAG (CARG) in adversarial settings:

Reduced ASR: SDAG significantly lowers the Attack Success Rate across all datasets, generators, and retrievers. For example, in single-document attacks on the NQ dataset, SDAG reduced ASR by more than half compared to CARG in several configurations.
Improved Accuracy: In most cases, SDAG also improves the accuracy of the generated answers compared to CARG, indicating that blocking cross-document attention preserves the integrity of benign information.
Robustness: The defense holds for both single-hop and multi-hop questions, demonstrating that SDAG does not hinder the model's ability to synthesize information across documents (verified in non-adversarial oracle settings).

2. Comparison with State-of-the-Art Defenses (RQ2)

Single-Document Attacks: SDAG outperforms existing state-of-the-art defenses (RAGDefender and Discern&Answer) in single-document attack scenarios. This is a critical finding, as many existing defenses rely on detecting similarities among multiple adversarial documents, a premise that fails when only one malicious document is present.
Multiple-Document Attacks: When SDAG is integrated with existing defenses (specifically RAGDefender), the combined approach establishes a new state-of-the-art, achieving statistically significant improvements in both ASR and accuracy over the baselines alone.

3. Spatial Positioning Analysis (RQ3)

The authors analyzed the relationship between the geometric distance of adversarial documents from benign documents in the embedding space and attack effectiveness:

Proximity Effect: Attacks are more effective when adversarial documents are "Near" (close to the centroid of benign documents) in the embedding space. "Far" attacks are less successful.
Defense Efficacy: The relative performance advantage of SDAG over CARG is even more pronounced when adversarial documents are geometrically distant from the benign set, suggesting that the block-sparse mechanism is particularly effective at isolating distinct semantic clusters.

Significance and Claims

The paper claims the following contributions and significance:

Identification of a Fundamental Flaw: The authors identify the standard causal attention mechanism as a primary vector for corpus poisoning, where cross-document interactions allow adversarial content to corrupt benign context.
A Novel, Lightweight Defense: SDAG offers a practical defense that requires no fine-tuning, no additional pipeline components, and minimal inference-time changes (mask modification only).
State-of-the-Art Performance: SDAG sets a new benchmark for RAG robustness, particularly in the under-explored single-document attack scenario. Its ability to integrate with other defenses to create a superior composite defense highlights its versatility.
Insight into Attack Dynamics: The analysis of embedding space positioning provides new insights into how attackers optimize their injections and how defenses can leverage geometric separation to improve robustness.

The authors note limitations, including the restriction to open-source models (due to the need for attention mask control), the focus on English Wikipedia-based QA, and the use of relatively smaller LLMs (7B-8B) for most experiments, though they note that larger models (70B) and reasoning-capable models also benefit from the approach.