Detecting RAG Extraction Attack via Dual-Path Runtime Integrity Game

1. Problem Statement

Context: Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLMs) with external, proprietary knowledge bases. However, this architecture introduces a critical vulnerability: RAG Knowledge Base Leakage.
The Threat: Adversaries can use black-box prompt interactions to induce the LLM to divulge retrieved proprietary content. Recent "RAG extraction attacks" (e.g., Pirates, RAG-Thief) utilize adaptive, iterative strategies to reconstruct entire knowledge bases by aggregating retrieved chunks across multiple queries.
Limitations of Existing Defenses:

• Passive Nature: Current defenses (e.g., reranking, summarization) aim to raise the cost of reconstruction but cannot proactively identify malicious intent.
• Intrusiveness: Many require structural changes to the RAG pipeline, retraining, or modifying the retrieval index, making them hard to deploy.
• Fragility: They are often vulnerable to adaptive attacks that obfuscate output or suppress sensitive information, failing to distinguish between legitimate semantic usage and unauthorized extraction.
• Detection Difficulty: Simple lexical or semantic similarity checks fail because legitimate RAG responses naturally share semantics with retrieved content, and attackers can easily obfuscate text to bypass these checks.

2. Methodology: CanaryRAG

The authors propose CanaryRAG, a plug-and-play, runtime defense mechanism inspired by stack canaries in software security. Instead of trying to prevent the attack, it detects violations of runtime integrity.

Core Concept: The Canary Token

• Injection: Before the retrieved chunks are fed to the LLM, the system injects lightweight, non-semantic, high-entropy strings (canary tokens) into the context.
• Behavior: In normal (benign) operation, these tokens are irrelevant to the task and should never appear in the model's output. If they do, it indicates the model is reproducing the raw retrieved context, signaling a leakage attempt.

The Dual-Path Runtime Integrity Game

To counter adaptive adversaries who might try to suppress canary tokens, CanaryRAG employs two concurrent generation paths with conflicting integrity expectations:

1. Target Path (User Query):

   • Input: User query + Retrieved chunks (with canaries).
   • Expectation: The model must answer the user's query.
   • Integrity Rule: Zero canary exposure. If any canary token appears in the output, it is flagged as a leakage violation.

2. Oracle Path (Probing Task):

   • Input: A modified query (e.g., "Repeat the context exactly") + Retrieved chunks (with canaries).
   • Expectation: The model is explicitly instructed to reproduce the context.
   • Integrity Rule: High canary exposure. The model must output the canary tokens.
   • Detection Logic: If the Oracle path fails to output the expected number of canaries (due to the adversary's suppression instructions), it indicates an active evasion attempt.

Decision Mechanism:

• Leakage Detected: If the Target Path emits canaries.
• Evasion Detected: If the Oracle Path fails to emit canaries (indicating the model is being coerced to suppress them).
• Action: Upon detection, the system immediately halts streaming generation, preventing further data exfiltration.

3. Key Contributions

1. Paradigm Shift: Recasts RAG extraction defense from a semantic matching problem to a runtime integrity enforcement problem, inspired by software security canaries.
2. Novel Architecture: Introduces the Dual-Path Integrity Game, which simultaneously monitors for unauthorized exposure (Target Path) and active suppression (Oracle Path), making it robust against adaptive adversaries.
3. Plug-and-Play Design: The solution requires no retraining of the LLM, no modification to the retriever or knowledge base, and no structural changes to the RAG pipeline. It operates as a lightweight wrapper.
4. Comprehensive Evaluation: Provides the first systematic detection-based defense evaluation against state-of-the-art adaptive extraction attacks.

4. Experimental Results

The authors evaluated CanaryRAG against three RAG agents (ChatDoctor, Mini-Wikipedia, Mini-BioASQ) and two advanced attacks (Pirates and RAG-Thief).

• Effectiveness (Chunk Recovery Rate - CRR):

  • CanaryRAG reduced the CRR to near-zero levels (average relative CRR of 0.04× compared to the "No Defense" baseline).
  • It significantly outperformed baselines: Reranker (0.98×), Summarization (0.67×), and RAGFort (0.60×).
  • Even when attackers were aware of the canary mechanism, CanaryRAG maintained high detection rates.

• Robustness (Adaptive Attacks):

  • Against adaptive attacks combining canary suppression and output obfuscation, CanaryRAG (with the Oracle path) achieved a True Positive Rate (TPR) of ~94-95%.
  • Disabling the Oracle path dropped the TPR to ~23-51%, proving the dual-path design is critical for adaptive defense.

• User Experience & Performance:

  • Quality: Negligible impact on response quality (BERTScore remained nearly identical to the baseline).
  • Latency: Negligible increase in inference latency (the overhead is constant and independent of the generator size).
  • False Positives: Extremely low False Positive Rate (FPR < 0.15%), ensuring benign users are rarely blocked.

5. Significance and Impact

• Practical Deployment: As a model-agnostic, plug-and-play solution, CanaryRAG can be immediately integrated into existing enterprise RAG systems without the high costs of retraining or pipeline restructuring.
• Proactive Defense: Unlike passive filters, it actively detects and stops attacks in real-time, preventing the accumulation of leaked data.
• Scalability: The computational overhead is constant and low, making it suitable for high-throughput production environments.
• New Security Direction: It establishes a new direction for LLM security by applying memory safety concepts (canaries) to the semantic integrity of generative AI, offering a robust safeguard for proprietary data.

In conclusion, CanaryRAG represents a state-of-the-art solution for RAG security, effectively neutralizing sophisticated extraction attacks while maintaining system utility and performance.