💻 computer science

Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generation

This survey provides a unified, pipeline-aware overview of Retrieval-Augmented Generation (RAG) robustness by formalizing threat models, categorizing attacks into accuracy, privacy, and fairness objectives, and reviewing stage-specific defenses, benchmarks, and explainability methods.

Original authors: Minh Tran, Cuong Dang, Tuc Nguyen, Khanh-Tung Tran, Minh Huynh Nguyen, Trinh Chau, Kien Le, Do Xuan Long, Jiahao Zhang, Hoang D. Nguyen, Thanh Le, Suhang Wang

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Minh Tran, Cuong Dang, Tuc Nguyen, Khanh-Tung Tran, Minh Huynh Nguyen, Trinh Chau, Kien Le, Do Xuan Long, Jiahao Zhang, Hoang D. Nguyen, Thanh Le, Suhang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern digital landscape, artificial intelligence has evolved from a simple chatbot into a sophisticated research assistant. These large language models can write code, summarize complex reports, and answer intricate questions. However, for years, they suffered from a persistent flaw: they often invented facts with absolute confidence. To fix this, engineers developed a system called Retrieval-Augmented Generation. Instead of relying solely on the information memorized during training, the system first searches a vast external library of documents, pulls out the most relevant facts, and then uses those facts to construct its answer. This approach grounds the AI in reality, making it far more accurate and less prone to making things up. Yet, this very mechanism of reaching out to the outside world introduces a new set of vulnerabilities. Just as a library can be sabotaged by someone slipping a forged book onto the shelves, the AI's external knowledge base can be poisoned, leading the system to retrieve false information and generate harmful or incorrect outputs.

A comprehensive new survey by researchers from institutions including the University of Science in Ho Chi Minh City and Pennsylvania State University maps out these emerging dangers and the defenses being built to counter them. The team did not just list attacks; they constructed a unified framework to understand how adversaries target the different stages of the AI's workflow. They found that attackers generally pursue three main goals: they want to make the AI give the wrong answer, they want to trick it into revealing private information, or they want to amplify unfair biases against specific groups of people. The researchers organized these threats by where they strike in the pipeline. The first point of failure is the search itself, where an attacker might inject misleading documents to ensure the AI finds the wrong source material. The second is the ranking stage, where the system decides which of the found documents are most important; here, an attacker might try to boost a fake document so it appears at the top of the list. The third is the generation stage, where the AI writes the final response, and the fourth is the traceback stage, which attempts to identify which document caused a mistake after the fact.

The survey reveals that the most common method of attack involves "corpus poisoning," where an adversary inserts malicious documents into the external database. These documents are crafted to look like legitimate sources but contain subtle errors or false claims. When the AI searches for an answer, it retrieves these poisoned documents and treats them as truth. For instance, an attacker could inject a document claiming that a specific antibiotic cures the flu, leading the AI to confidently give dangerous medical advice. Another sophisticated tactic is the "backdoor attack," where the system is trained or manipulated to react only when a specific, hidden trigger phrase is present. If a user asks a question containing that trigger, the AI ignores all other evidence and retrieves a pre-planted, malicious document to generate a harmful response. The researchers also detailed "prompt injection" attacks, where malicious instructions are hidden inside a retrieved document. Once the AI reads this document, it might interpret the hidden text as a command to ignore the user's original question and instead reveal sensitive data or perform an unauthorized action.

Beyond simply breaking the system, the survey highlights how these attacks can compromise privacy and fairness. In terms of privacy, attackers can use the AI as a tool to extract confidential information from the database. By asking carefully crafted questions, they can trick the system into revealing private details about individuals or organizations that were stored in the documents. One study cited in the survey showed that attackers could achieve a success rate of up to 94% in leaking specific documents when using a backdoor trigger. Regarding fairness, the researchers found that attackers can poison the database with biased content to skew the AI's output against certain demographic groups. For example, by injecting documents that associate specific genders with negative traits, the AI could be manipulated to generate responses that reinforce harmful stereotypes, effectively amplifying societal biases through the lens of the machine.

To combat these threats, the researchers reviewed a wide array of defense strategies, each designed for a specific stage of the process. At the retrieval stage, defenses focus on cleaning the database and making the search engine more resistant to manipulation, such as by filtering out low-quality sources or detecting unusual patterns in the text. During the reranking phase, systems are being developed to cross-check multiple documents and demote those that seem suspicious or inconsistent with the rest of the evidence. At the generation stage, the AI itself is being taught to be more skeptical, learning to identify when a retrieved document conflicts with known facts or when the text contains hidden instructions. Finally, in the traceback stage, new methods are emerging that can trace a wrong answer back to the specific document that caused it, allowing developers to pinpoint and remove the source of the error.

Despite these advances, the survey concludes that the field is still in its early stages and faces significant hurdles. Many current defense methods rely on having full access to the internal workings of the AI system, which is rarely the case in real-world applications where the system is a "black box." Furthermore, there is a constant trade-off between the effectiveness of an attack and how obvious it is; highly effective attacks often produce text that sounds unnatural, making them easier to spot, while stealthy attacks are harder to execute. The researchers also noted that as AI systems evolve to handle more complex tasks, such as analyzing graphs of data or processing images alongside text, new and unpredictable vulnerabilities are likely to emerge. The path forward requires a continuous cycle of testing and improvement, ensuring that as these powerful tools become more integrated into our lives, they remain robust against those who would seek to exploit them.

Technical Summary: Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generation

Problem Statement

Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLMs) by grounding outputs in external knowledge, aiming to improve factual accuracy and reduce hallucinations. However, the introduction of a retrieval pipeline creates new, complex attack surfaces distinct from standalone LLMs. Attackers can exploit the pipeline to infer private corpus content, inject misleading evidence to induce incorrect outputs, or amplify demographic biases. While existing literature addresses RAG security, current surveys are often fragmented, lacking a unified framework that connects attacker objectives, threat models, and stage-specific defenses across the entire RAG pipeline (retrieval, reranking, generation, and traceback). Furthermore, emerging RAG variants (e.g., GraphRAG, Multimodal RAG) introduce structural constraints that reshape attack surfaces but remain under-explored in security contexts.

Methodology

This survey employs a systematic methodology to construct a unified, pipeline-aware overview of RAG robustness:

Literature Collection: The authors identified seed papers (e.g., PoisonedRAG, TrojanRAG, Neural Exec, The Good and the Bad) and expanded the dataset via citation tracking and keyword searches (retrieval, attack, defense, privacy, robustness).
Taxonomy Design: The survey is structured around two critical questions: what an attacker aims to compromise (objective) and where the failure occurs in the pipeline (stage).
- Attacks are organized by objective: Accuracy, Privacy, and Fairness.
- Defenses are organized by pipeline stage: Retrieval, Rerank, Generation, and Traceback.
Threat Modeling: The authors formalize threat models based on attacker goals (targeted vs. untargeted), knowledge (black-box, gray-box, white-box), and capabilities (insert-only access to corpus, query access to retriever/generator).
Scope Expansion: The survey extends beyond standard RAG to include variants like GraphRAG, Multimodal RAG, and Agentic RAG, analyzing their specific security implications.

Key Contributions

1. Unified Threat Model and Attack Taxonomy

The paper formalizes threat models across the three primary RAG components: the Corpus ( $D$ ), the Retriever ( $R$ ), and the Generator ( $F$ ). Attacks are categorized into three primary objectives:

Accuracy Attacks: Aim to induce incorrect or harmful outputs.
- Targeted: Induce specific wrong answers via Corpus Poisoning (injecting misleading documents), Backdoor Attacks (activating malicious behavior via triggers), or Prompt Injection (embedding executable instructions in retrieved docs).
- Untargeted: Broadly degrade system reliability or bypass safety constraints across diverse queries.
Fairness Attacks: Induce or amplify systematic disparities (e.g., gender, race) by injecting biased evidence or poisoning query encoders to align triggers with specific demographic groups.
Privacy Attacks: Extract sensitive information without altering system outputs.
- Targeted: Membership Inference (determining if a doc is in the corpus), Privacy Extraction (forcing the model to reveal retrieved private content), and Confused-Deputy behaviors (tricking over-privileged entities).
- Untargeted: Large-scale extraction of hidden corpus data via probing queries.

2. Pipeline-Aware Defense Taxonomy

Defenses are categorized by the stage at which they intervene:

Retrieval-Stage: Focuses on Robust Retrieval (diagnostic resources, evidence-centric retrieval, corpus sanitization) and Robust Training (hard negative training, low-level perturbation training) to improve the initial evidence set.
Rerank-Stage: Refines the top- $k$ candidates by promoting benign documents and suppressing suspicious ones using cross-document reasoning and similarity graphs (e.g., GRADA).
Generation-Stage: Operates at the LLM level, divided into:
- Proactive: Filtering (consistency-based, influence-based, mechanistic) and detection mechanisms to flag poisoned inputs before generation.
- Passive: Robust reasoning (isolation-based generation, conflict-aware reasoning) and robust training (adversarial training, alignment, self-rationale training) to mitigate influence during decoding.
Traceback-Stage: Applied post-failure to attribute responsibility. Includes context-level, document-level, and dataset-level attribution to identify the specific sources of misleading or unsafe outputs.

3. Analysis of Emerging RAG Variants

The survey highlights how new architectures alter the threat landscape:

GraphRAG: While graph indexing may reduce the effectiveness of standard corpus poisoning, it introduces new risks like entity/relational poisoning and privacy extraction via subgraph reconstruction.
Multimodal RAG: Expands attack surfaces to include image-text pairs, enabling corpus poisoning and membership inference across modalities.
Agentic RAG: Introduces stateful loops where attacks can persist in memory, manipulate tool selection, or exploit untrusted tool observations, shifting the security objective from answer integrity to the integrity of the entire retrieve-reason-act-update loop.

4. Benchmarks and Explainability

The paper summarizes robustness benchmarks (e.g., SafeRAG, URAG) that evaluate failure modes like corpus poisoning, retrieval brittleness, and knowledge conflicts. It also reviews explainability methods (attribution, mechanistic, structural, and system-level) that reveal why poisoned context influences model behavior, such as the preference for coherent evidence chains or the shift from internal safety policies to external evidence synthesis.

Results and Findings

Vulnerability: RAG systems are highly susceptible to corpus poisoning and prompt injection, where malicious documents can steer generators toward specific targets or leak private data.
Defense Limitations: Current defenses are often fragmented. Many rely on strong white-box assumptions that do not translate to black-box deployments. Traceback methods are largely diagnostic rather than preventive.
Trade-offs: There is a tension between attack stealthiness and effectiveness; optimized adversarial texts often lack semantic naturalness, making them detectable. Similarly, universal attacks improve scalability but may lose precision compared to query-specific optimizations.
Emerging Risks: Privacy risks in GraphRAG and Multimodal RAG are underexplored, and Agentic RAG introduces severe consequences where compromised states lead to unauthorized external actions.

Significance and Claims

The authors position this work as a roadmap for developing trustworthy RAG systems. Its significance lies in:

Unification: Providing a comprehensive, pipeline-aware taxonomy that connects attack objectives with specific defense stages, addressing gaps in prior surveys where existing works provided comprehensive coverage of key dimensions (e.g., factuality, robustness) but offered limited analysis of attack structures or adversarial defenses.
Comprehensiveness: Covering not just standard RAG but also emerging variants (GraphRAG, Agentic RAG) and integrating benchmarks and explainability methods.
Future Direction: Highlighting open challenges, such as the lack of transferability of attacks to black-box settings, the need for stealthier attacks, and the necessity of fine-grained source attribution for privacy leakage.

The paper concludes by committing to a public repository to continuously track developments in this rapidly evolving field, emphasizing that building robust RAG systems requires a holistic understanding of the entire retrieval-augmented pipeline.