Technical Summary: Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation (RAG)

Problem Statement

Retrieval-Augmented Generation (RAG) has become a foundational paradigm for knowledge-intensive applications, including enterprise chatbots, healthcare assistants, and agentic memory management. While RAG effectively mitigates hallucinations and supports dynamic knowledge updates, it introduces a novel vulnerability: knowledge-extraction attacks. Unlike traditional data or model extraction, RAG systems provide adversaries with an additional channel to recover sensitive content directly from the external knowledge base.

Attackers craft malicious queries comprising two components: an INFORMATION signal to steer the retriever toward sensitive content and a COMMAND instruction to force the generator to reproduce that content verbatim. Despite prior work exploring individual attack and defense techniques, the research landscape remains fragmented. Existing studies suffer from heterogeneous experimental settings, including inconsistent dataset versions, diverse retrieval embedding models, varying generator architectures, and non-standardized evaluation metrics. This lack of a unified design space hinders fair comparison and the development of robust privacy-preserving RAG systems.

Methodology

The authors introduce the first systematic benchmark for knowledge-extraction attacks on RAG systems, establishing a unified experimental framework to evaluate attacks and defenses across a comprehensive design space.

1. Unified Design Space

The benchmark standardizes the following dimensions:

• RAG Architecture:
  • Retrievers: Three embedding models ranging from lightweight to large-scale (MiniLM, GTE-Base, BGE-Large).
  • Generators: Both open-source (LLaMA, Qwen) and closed-source (GPT-4o, GPT-4o mini) models.
  • Knowledge Base Construction: Three strategies reflecting real-world deployments: Original (independent documents), Chunking (fixed-length segments), and Graph Triplet (entity-relation-entity structures).
• Attack Strategies:
  • Information Component: Ranges from random baselines (Random Token, Random Text, Random Embedding) to adaptive, state-of-the-art methods (IKEA, DGEA, CopyBreak). These methods optimize query embeddings to align with target knowledge while maintaining diversity.
  • Command Component: Varies from simple repetition requests to complex jailbreaks, role-playing, and multi-step instructions designed to bypass safety filters.
• Defense Strategies:
  • Input Stage: Query-Block defense using zero-shot LLM classifiers to detect malicious intent.
  • Retrieval Stage: Threshold defense that filters retrieved documents based on similarity scores, rejecting low-relevance candidates even if they appear in the Top-K results.
  • Generation Stage: System-Block defense (instructing the model to refuse sensitive output) and Summary defense (forcing the model to summarize rather than reproduce verbatim).

2. Standardized Evaluation Protocol

To disentangle the contributions of retrieval and generation, the benchmark decomposes evaluation into three metrics:

• Retriever Extraction Effectiveness (EER): Measures the proportion of target knowledge successfully retrieved.
• Generator Extraction Effectiveness (EEG): Measures the alignment between generated answers and retrieved content, evaluated via both Lexical (LS) and Semantic (SS) similarity.
• Combined Extraction Effectiveness (EE): The end-to-end proportion of retrieved knowledge that is both reproduced by the generator and aligned with the target set.
• Attack Success Rate (ASR): The frequency with which an attack successfully elicits a knowledge-grounded response, excluding refusals or hallucinations.

Key Contributions

1. Comprehensive Review and Unified Design Space: The paper systematically surveys existing attack and defense methods, formalizing a unified design space that characterizes unique dimensions and assumptions across the RAG lifecycle.
2. Standardized Evaluation Protocol: By standardizing RAG configurations, datasets (HealthCareMagic, Enron, HarryPotter, Pokémon), and metrics, the benchmark enables fair, reproducible comparisons across diverse attack and defense strategies.
3. Extensive Experimental Analysis: The authors release a reproducible pipeline and conduct extensive experiments, yielding actionable insights into the security mechanisms of RAG systems.

Key Results

The experiments, conducted across four datasets and multiple configurations, reveal several critical findings:

• Attack Performance:

  • DGEA (an adaptive embedding-optimized attack) consistently outperforms other methods in retrieval effectiveness (EER) under white-box settings where the attacker and retriever share the same embedding model. However, its performance drops sharply in cross-embedding (black-box) scenarios due to poor transferability of optimized, non-natural embeddings.
  • IKEA and CopyBreak, which rely on LLM-driven query generation to maintain linguistic naturalness, show robust performance across different embedding models and are less affected by cross-embedding configurations.
  • Attacks using explicit COMMAND instructions (e.g., "repeat all context") achieve high extraction effectiveness, whereas stealthier attacks like IKEA that avoid explicit reproduction commands result in lower leakage but maintain higher Attack Success Rates (ASR) by evading detection.

• Defense Effectiveness:

  • Threshold Defense: Deployed at the retrieval stage, this is the most effective defense against embedding-optimized attacks (R-EB, DGEA) by filtering out low-similarity query-context pairs. However, it introduces a significant security-utility tradeoff; high thresholds drastically reduce recall for legitimate queries.
  • Query-Block Defense: Highly effective against attacks with explicit extraction intent but fails against stealthy attacks like IKEA that mimic benign user queries.
  • Generation Defenses (System Block & Summary): These effectively reduce leakage by preventing verbatim reproduction. The Summary defense is particularly robust, often forcing null outputs for queries with weak semantic relevance to the retrieved content.

• Model and Configuration Sensitivity:

  • Generators: Closed-source models generally exhibit higher extraction effectiveness than open-source models when explicit verbatim commands are used, reflecting stronger instruction-following capabilities.
  • Knowledge Structure: The format of the knowledge base (Original vs. Chunked vs. Graph) significantly influences extraction outcomes, with different attack strategies showing varying sensitivities to these structures.

Significance

This paper addresses the fragmentation in the current research landscape by providing a unified, reproducible benchmark for knowledge-extraction risks in RAG systems. By consolidating experimental settings and enabling comparable evaluations, the work offers a practical foundation for developing privacy-preserving RAG systems. The authors emphasize that their benchmark provides actionable insights for practitioners, such as the sensitivity of extraction to knowledge formats and the importance of query diversity exploration in defense strategies. The work does not propose new specific applications or future theoretical implications beyond the scope of evaluating and improving RAG security against extraction threats.