💬 NLP

Tracing Target Answers in Poisoned Retrieval Corpora via Token Influence Attribution

This paper introduces TRACE, a lightweight framework that detects corpus poisoning attacks in Retrieval-Augmented Generation systems by attributing token influence to identify malicious documents and trace target answers without requiring auxiliary classifiers or significant computational overhead.

Original authors: Yan-Lun Chen, Pin-Yu Chen, Chia-Mu Yu, Ying-Dar Lin, Yu-Sung Wu, Wei-Bin Lee

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yan-Lun Chen, Pin-Yu Chen, Chia-Mu Yu, Ying-Dar Lin, Yu-Sung Wu, Wei-Bin Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Fake News" Library

Imagine you have a very smart librarian (the AI) who knows a lot, but sometimes forgets details. To help, you give them a stack of reference books (the Retrieval Corpus) to look at before answering your questions.

This is how RAG (Retrieval-Augmented Generation) systems work. They are great for businesses and customer service.

The Attack: A bad actor (the Poisoner) sneaks into the library and plants several fake books. These books all tell the same lie. For example, if you ask, "At what age does Medicare start?" the real books say 65, but the fake books all say 50. Because the AI trusts the books it finds, it might start believing the lie and tell you the wrong answer.

The Old Way to Catch It: Previous security tools tried to catch these fake books by hiring a second, expensive librarian to check every single book, or by training a special "lie detector" robot. This takes a lot of time, money, and computer power.

The Solution: TRACE (The "Influence Detective")

The authors introduce TRACE, a lightweight, fast, and cheap way to catch these poisoned books without hiring extra help.

Instead of asking, "Is this book fake?" TRACE asks a different question: "Which specific words in this book are shouting the loudest to the AI?"

Think of it like a magnet test.

The Magnet: The AI has a natural tendency to be pulled toward certain words when it's trying to give an answer.
The Test: TRACE looks at the books the AI is reading. It calculates a "magnetic pull" score for every single word.
The Clue: If the AI is being tricked, it will be weirdly obsessed with specific words (like the number "50" in our Medicare example) across many different fake books.

How TRACE Works (The Two-Step Dance)

Step 1: The "Who's Shouting?" Scan (Keyword Searching)

TRACE reads all the books the AI found.
It ignores boring words like "the," "and," or "is" (because those don't change the answer).
It looks for the words that have the strongest "magnetic pull" on the AI's brain.
If the word "50" keeps showing up as the strongest pull in multiple different books, TRACE flags it as a Suspicious Keyword.

Step 2: The "Double Check" (Secondary Verification)

Now that TRACE has a list of suspicious words (like "50"), it does a second test.
It pretends the AI is trying to say "Yes, the answer is: 50."
It checks again: Do these specific words still have that super-strong magnetic pull in the books?
The Verdict: If the same suspicious words keep popping up as the most influential across the whole stack of books, TRACE sounds the alarm: "Poison Detected!"

Why This is a Big Deal

The paper claims TRACE is special for three reasons:

It's Lightweight: It doesn't need a second AI or a special training robot. It just uses the math inside the AI it's already protecting. It's like checking a book for fingerprints instead of hiring a whole new police force.
It's Fast: It can run right before the AI gives an answer, catching the lie in real-time.
It Reveals the Lie: Not only does it say "This is a fake book," but it also points a finger at the specific lie. In our example, it doesn't just say "Danger," it says, "The attacker is trying to make you believe the answer is 50."

The Results (The Scoreboard)

The authors tested TRACE on six different types of "smart librarians" (AI models) and three different question sets.

Detection: It caught the poisoned books more than 90% of the time.
False Alarms: It rarely cried "wolf" when the books were actually clean (less than 10% false alarms).
Speed: It was significantly faster than the previous best method (RAGForensics), taking about 1 second per question compared to 9 seconds for the old method.

The Bottom Line

TRACE is a clever, low-cost security guard for AI systems. Instead of guessing if a document is bad, it traces the "footprints" of the attacker's specific words. If it sees the same suspicious words leading the AI astray in multiple documents, it stops the AI from giving the wrong answer and tells you exactly what the attacker tried to hide.

Technical Summary: Tracing Target Answers in Poisoned Retrieval Corpora via Token Influence Attribution

Problem Statement

Retrieval-Augmented Generation (RAG) systems, while enhancing Large Language Models (LLMs) with external knowledge, are vulnerable to corpus poisoning attacks. In these attacks, adversaries inject misleading documents into the retrieval corpus to manipulate the LLM's output toward a specific, erroneous target answer. Existing detection methods typically rely on auxiliary classifiers or additional LLM-based verification (e.g., querying a separate model to verify outputs). These approaches introduce substantial computational overhead and maintenance costs, making them impractical for many enterprise deployments that require lightweight, real-time defense mechanisms.

Methodology: TRACE Framework

The authors propose TRACE, a lightweight detection framework that identifies poisoning attacks by tracing answer-related tokens through token influence attribution. Unlike methods that classify documents as benign or malicious, TRACE analyzes the internal mechanics of the target LLM to identify tokens exerting the strongest influence on predictions. The framework operates in two distinct phases:

Phase 1: Keyword Searching

Influence Calculation: For each retrieved document, the system computes token-level influence scores via backpropagation. The target string is a predefined affirmation (e.g., "Yes, the answer is:"). The influence of a token embedding $x_i$ is calculated as the squared norm of the gradient of the log-probability of the affirmation with respect to that token: $\|\nabla_{x_i} \log P_\theta(y|x_{1:n})\|^2$ .
Filtering: The system extracts the top- $k_1$ most influential tokens, strictly excluding punctuation, frequent words (stopwords), and tokens already present in the user query to reduce noise.
Phrase Construction: Contiguous high-influence indices are concatenated into candidate phrases.
Consensus Filtering: These phrases are aggregated across all retrieved documents. A phrase is admitted into the candidate keyword set $K$ only if it recurs in a number of documents meeting a threshold $a_1$ . This isolates the consolidated malicious payload synchronized across the poisoned corpus.

Phase 2: Secondary Verification

Targeted Re-evaluation: For each candidate keyword $p \in K$ , the system re-evaluates the retrieved documents. Unlike Phase 1, the candidate phrase $p$ itself is set as the target prediction to compute new token-level influence gradients.
Trigger Identification: The algorithm identifies the top- $k_2$ influential tokens within each document that correspond to the candidate phrase.
Final Verification: These tokens are aggregated, and a token is added to the final target set $T$ only if it appears as a high-influence trigger in at least $a_2$ separate documents.
Detection Flag: If the final target set $T$ is non-empty, the retrieved corpus is flagged as poisoned.

Key Contributions

Lightweight Framework: TRACE introduces a detection mechanism that requires no auxiliary classifiers and no additional LLM verification, significantly reducing computational overhead.
Token-Attribution Paradigm: The paper proposes a novel detection approach that identifies poisoning by tracing recurrent high-influence tokens across retrieved documents, directly linking token influence to model predictions.
Target Answer Revelation: Beyond mere detection, TRACE demonstrates the ability to uncover attacker-specified target answers with low computational cost, providing actionable intelligence for defensive analysis.

Experimental Results

The authors evaluated TRACE on three QA benchmarks (Natural Questions, HotpotQA, MS-MARCO) using six diverse LLMs (Llama-3.1-8B, Gemma-3-4b, Qwen-3.5-4B, Vicuna-7b, Mistral-7B, and Phi-4-mini) against the PoisonedRAG attack baseline.

Detection Performance: TRACE consistently achieved a True Positive Rate (TPR) of over 90% across nearly all evaluated models. On the Phi-4-mini model, it secured an average TPR of 98.87%.
False Positive Control: By tuning hyperparameters, the framework suppressed the False Positive Rate (FPR) below 10% in benign scenarios, demonstrating robust control over false alarms.
Target Answer Identification: The framework successfully identified the attacker's target answer in approximately 90% of cases on the NQ and MS-MARCO datasets, and over 70% on HotpotQA.
Efficiency: Compared to RAGForensics (a baseline using LLM-based verification), TRACE achieved superior detection performance with significantly lower execution time (e.g., ~1.0s vs ~9.0s per question for Llama-3.1-8B).

Significance and Claims

The paper claims that TRACE offers a real-time, first-line defense for RAG pipelines. Its significance lies in:

Zero Prior Knowledge: Unlike reactive defenses that require knowing the target answer to verify outputs, TRACE identifies target answer tokens before generation without prior knowledge of the attack.
Operational Viability: By eliminating the need for training auxiliary models or repeated LLM inference, TRACE addresses the practical constraints of enterprise deployment.
Transparency: The ability to reveal the specific tokens driving the malicious prediction allows organizations to intercept malicious outputs and mitigate reputational risks.

The authors acknowledge limitations, noting that the method relies on the statistical recurrence of target answers and may be less sensitive to attacks involving very few poisoned documents. However, they argue that such sparse injections inherently reduce the attack's success rate (ASR), making TRACE an effective filter for high-impact poisoning attempts. The work is presented strictly as a defensive tool for security evaluation, with no new attack mechanisms or datasets released.