🤖 AI

Through the Stealth Lens: Attention-Aware Defenses Against Poisoning in RAG

This paper proposes an attention-aware defense mechanism using Normalized Passage Attention Scores and an Attention-Variance Filter to detect and mitigate non-stealthy poisoning attacks in Retrieval-Augmented Generation systems, while also highlighting the inherent difficulty of achieving true stealth in such attacks.

Original authors: Sarthak Choudhary, Nils Palumbo, Ashish Hooda, Krishnamurthy Dj Dvijotham, Somesh Jha

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sarthak Choudhary, Nils Palumbo, Ashish Hooda, Krishnamurthy Dj Dvijotham, Somesh Jha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Smart Librarian" Problem

Imagine you have a brilliant, super-smart librarian (the LLM) who knows a lot but sometimes makes things up or forgets recent news. To fix this, you give the librarian a stack of reference books (the Retrieved Passages) to look at before answering a question. This system is called RAG (Retrieval-Augmented Generation).

The problem? A bad actor (the Attacker) can sneak a fake, misleading book into that stack. Even if they only slip in one fake book among ten real ones, the librarian might get tricked into reading that fake book more closely than the others and give you the wrong answer.

The Core Discovery: "The Loud Whisper"

The authors of this paper discovered something surprising: Current attacks aren't actually stealthy.

Think of the librarian's brain as having a spotlight. When the librarian reads the stack of books to answer a question, the spotlight naturally shines on the most important words.

In a normal situation: The spotlight is shared fairly evenly among all the real books.
In a poisoning attack: To trick the librarian, the fake book has to shout its answer very loudly. Because it's shouting so hard to override the truth, the librarian's spotlight gets "stuck" on that one fake book, ignoring the others.

The paper calls this a "Loud Whisper." The attack tries to be quiet, but because it has to work so hard to change the answer, it leaves a huge, glowing footprint in the librarian's attention.

The New Defense: The "Spotlight Meter" (AV Filter)

The researchers built a new security tool called the Attention-Variance Filter (AV Filter). Here is how it works, using a simple analogy:

Imagine you are a security guard watching a group of people (the books) trying to convince a judge (the librarian) of a story.

The Check: The guard measures how much "attention" (spotlight) each person gets from the judge.
The Pattern: In a healthy group, everyone gets a roughly equal amount of attention.
The Alarm: If one person is getting 80% of the spotlight while everyone else gets 5%, the guard knows something is wrong. That person is likely the "poisoned" book trying to hijack the conversation.
The Action: The guard kicks that loud, suspicious person out of the room before the judge makes a final decision.

This tool is called the Normalized Passage Attention Score (NPAS). It calculates exactly how much the librarian is focusing on each book. If the focus is too uneven (high variance), the system removes the suspicious book.

The Results: Winning the Game

The paper tested this against several types of attacks (like "PoisonedRAG" and "Prompt Injection").

Detection: The new filter caught the bad books about 78% of the time, even when the attackers tried to hide them.
Protection: When using this filter, the system got the right answer much more often (up to 20% better) than previous security methods, without slowing things down or making mistakes on normal questions.

The "Cat and Mouse" Game: Adaptive Attacks

The researchers also asked: "Can the bad actors learn to be quieter?"

They created a new type of attack called an Adaptive Attack. This is like a spy who knows the security guard is watching the spotlight. The spy tries to write the fake book in a way that spreads the spotlight out more evenly, making it look like a normal book.

Did it work? Partially. These smarter attacks managed to trick the filter about 35% of the time.
The Catch: To do this, the spy had to spend a massive amount of time and computing power (thousands of times more than a normal attack) to craft the perfect fake book for each specific question.

The Conclusion: While it is possible to make attacks slightly sneakier, doing so is incredibly expensive and difficult. The "stealth" of these attacks is not free; it comes with a huge cost.

Summary of Limitations (What the paper doesn't claim)

Majority Rule: If the bad guys manage to sneak in more than half of the books (e.g., 6 fake books out of 10), this filter won't work. It relies on the good books being the majority.
Specific Targets: This works best for questions with clear answers (like "How many towers are built?"). It might not work as well if the attack is trying to change the style of the writing or steal private data, rather than changing the factual answer.

The Bottom Line

The paper proves that trying to poison a smart librarian's reference stack leaves a "glowing fingerprint" in the librarian's attention. By measuring that fingerprint, we can easily spot and remove the fake books, keeping the answers accurate without needing to rebuild the whole library.

Technical Summary: Through the Stealth Lens: Attention-Aware Defenses Against Poisoning in RAG

Problem Statement

Retrieval-Augmented Generation (RAG) systems, which integrate Large Language Models (LLMs) with external knowledge bases, are vulnerable to retrieval corruption attacks. In these attacks, adversaries inject poisoned passages into the knowledge database (e.g., via manipulated Wikipedia pages or social media) to corrupt the retrieved context. Even with low corruption rates (e.g., one poisoned passage among ten), existing attacks like PoisonedRAG can successfully manipulate the LLM's response to a target answer.

The core challenge addressed in this paper is stealth. While existing attacks achieve high success rates, the authors argue they are not designed to be stealthy. To override the consensus of benign passages, adversarial passages must disproportionately influence the LLM's inference process. This inherent requirement creates a detectable "malicious trace" in the model's internal states, specifically in the distribution of attention weights. The paper posits that current defenses often overlook these intermediate signals, relying instead on surface-level semantics or perplexity, which are less effective against sophisticated poisoning.

Methodology

1. Formalizing Stealth: The SADG Game

The authors formalize the concept of stealth through the Stealth Attack Distinguishability Game (SADG). In this security game, a defender attempts to distinguish between a benign retrieved set and a corrupted one (containing up to $\epsilon$ fraction of poisoned passages). An attack is considered $\tau$ -stealthy if the defender's advantage in distinguishing the sets is at most $\tau$ . The paper argues that successful low-corruption attacks inherently fail this test because they concentrate influence on a small subset of passages, creating a detectable statistical anomaly.

2. Normalized Passage Attention Score (NPAS)

To quantify the influence of individual passages on the generated response, the paper introduces the Normalized Passage Attention Score (NPAS).

Mechanism: The method aggregates attention weights from the LLM's decoder layers and heads. For a retrieved passage $z_t$ , it sums the attention scores from all generated response tokens to the top- $\alpha$ most attended tokens within $z_t$ .
Normalization: The score is normalized by the total attention across all $k$ retrieved passages to enable comparison across different queries and models.
Rationale: In benign scenarios, attention is distributed relatively uniformly across relevant passages. In successful poisoning attacks, the model concentrates attention heavily on the poisoned passage (specifically tokens matching the adversarial target), resulting in a high-variance distribution of NPAS.

3. The Attention-Variance Filter (AV Filter)

Leveraging the observation that poisoned sets exhibit high variance in NPAS, the authors propose the Attention-Variance Filter (AV Filter).

Process: The filter iteratively removes the passage with the highest NPAS from the retrieved set.
Stopping Condition: The process continues until either the variance of the remaining NPAS scores drops below a threshold $\delta$ or the number of removed passages reaches the corruption budget $\epsilon$ .
Implementation: To mitigate positional bias (recency effects), passages are reordered by their NPAS before filtering. The threshold $\delta$ is estimated from clean data (mean + 1 standard deviation).

4. Adaptive Attacks

To test the limits of this defense, the authors develop adaptive attacks that optimize for stealth. Using jailbreak methodologies (extending GCG and AutoDAN), these attacks minimize a composite loss function:
$L_t = L_1 + \lambda \cdot L_2$
Where $L_1$ is the cross-entropy loss for the target response (effectiveness) and $L_2$ is the variance of NPAS (stealth). This forces the attacker to find poisoned passages that induce the target answer without creating a high-variance attention signature.

Key Contributions

Formalization of Stealth: Introduction of the SADG to rigorously define and measure the stealth of RAG poisoning attacks.
NPAS Metric: Development of the Normalized Passage Attention Score to quantify passage-to-response influence, demonstrating that it becomes skewed under corruption.
AV Filter Defense: Proposal of a lightweight, attention-variance-based filter that reliably distinguishes and removes corrupted passages without altering the inference pipeline.
Comprehensive Evaluation: Extensive experiments across 4 datasets, 5 LLMs, and 5 attack types, including the development of adaptive attacks to probe the defense's limits.
White/Black-Box Applicability: Demonstration that the defense works in white-box settings (direct access to model internals) and black-box settings (using an auxiliary open-source model to approximate attention signals).

Results

Detection Capability (RQ1): The defender utilizing NPAS variance ( $D_{AV}$ ) successfully identifies corrupted sets in existing attacks with an average win rate of 0.78 in the SADG, significantly outperforming random guessing.
Robustness (RQ2): The AV Filter significantly improves robust accuracy (RACC) compared to baselines.
- It achieves up to 23% higher accuracy than baseline defenses in benign settings.
- Under attack, it yields up to 20% higher RACC than baselines like Certified Robust RAG (Keyword/Decoding).
- It reduces the Attack Success Rate (ASR) of vanilla RAG (which can reach ~88% on some datasets) to an average of 6.6% on RQA-MC, comparable to or better than existing robust methods.
Adaptive Attack Limits (RQ3): Adaptive attacks can bypass the AV Filter, achieving an ASR of up to 35%. However, this comes at a prohibitive cost:
- They require full access to model internals and the knowledge base.
- They involve query-specific optimization, increasing runtime by approximately $10^3$ to $10^4$ times compared to standard attacks (e.g., ~18,000 seconds per query).
- The authors conclude these represent an extreme theoretical limit rather than a practical threat.

Significance and Claims

The paper claims that stealth and effectiveness are fundamentally trade-offs in RAG poisoning. To successfully manipulate a response with a minority of poisoned passages, an adversary must exert disproportionate influence, which leaves a detectable signature in the attention weights.

Defense Efficacy: The AV Filter offers a practical, low-overhead defense that preserves the utility of RAG systems (minimal drop in clean accuracy) while effectively mitigating current low-stealth attacks.
Limitations: The defense relies on specific assumptions: the corruption must be a minority ( $\epsilon < 0.5$ ), correct knowledge must be redundantly represented in benign passages, and the attack must target response content (not style or data leakage).
Future Outlook: While adaptive attacks show that stealth can be improved, they remain computationally expensive and resource-intensive. The paper suggests that achieving true, efficient stealth in RAG poisoning remains a significant open challenge, and polynomial-time defenses like the AV Filter are currently sufficient for practical scenarios.

The authors emphasize that their dual contribution (defense and adaptive attack) aims to drive the development of more robust RAG systems by exposing the current vulnerabilities and the high cost of overcoming them.