Technical Summary: A Layered Security Framework Against Prompt Injection in RAG-Based Chatbots

1. Problem Statement

Prompt injection is identified as the most critical vulnerability in Large Language Model (LLM) deployments, particularly within Retrieval-Augmented Generation (RAG) architectures. Unlike traditional software injection attacks that exploit parsing bugs, prompt injection exploits the fundamental architectural property of LLMs: the inability to distinguish between trusted instructions (system prompts) and untrusted data (user inputs or retrieved documents) within a single token stream.

Existing defenses are insufficient because they operate at isolated stages of the inference pipeline:

• Input filters cannot inspect retrieved documents, leaving systems vulnerable to indirect prompt injection (where malicious payloads are embedded in knowledge base documents).
• Output monitors cannot prevent malicious payloads from reaching the model in the first place.
• Instruction hierarchy approaches often require model fine-tuning or rely on structural signals that models may not reliably honor.

Consequently, RAG-based chatbots remain vulnerable to both direct injection (via user input) and indirect injection (via poisoned knowledge base documents), leading to risks such as persona hijacking, policy bypass, data exfiltration, and the generation of harmful content.

2. Methodology: The Three-Layer Framework

The authors propose a model-agnostic, three-layer defense framework designed to intercept attacks at distinct points in the inference pipeline without modifying the underlying LLM. The system operates as middleware.

Layer 1: Input Screening

This layer operates on raw user input before retrieval or context assembly. It employs two complementary mechanisms:

• Pattern-Based Detection: A rule engine matches normalized input against a curated library of injection signatures (e.g., "ignore previous instructions," "repeat system prompt"). It handles obfuscation techniques like Base64 encoding and URL encoding.
• Semantic Anomaly Detection: A fine-tuned semantic classifier (using a sentence-transformer backbone) scores inputs for adversarial intent. This captures novel phrasings and paraphrased attacks that evade rule-based signatures.
• Disposition: Inputs are either passed (with an anomaly flag), sanitized (matching segments replaced), or blocked.

Layer 2: Privilege-Constrained Context Assembly

This layer addresses the root cause of injection: the flat context window. It enforces an explicit instruction hierarchy during context assembly, prior to LLM inference:

• Privilege Tiers:
  1. Operator Tier (Highest): System prompt instructions.
  2. Retrieval Tier (Intermediate): Retrieved documents (treated as reference data, not executable instructions).
  3. User Tier (Lowest): User queries.
• Mechanism: Context segments are annotated with provenance tags ([SYS], [RET], [USR]). A meta-instruction is prepended to the prompt, instructing the model to refuse directives from lower-tier sources that conflict with higher-tier instructions.
• Scanning: Retrieved document chunks are scanned for explicit override patterns before being admitted to the context.

Layer 3: Output Auditing

This layer serves as the final safety net, auditing the model's response before delivery:

• Policy Rule Engine: Checks the output against specific predicates, including:
  • System Prompt Similarity: Cosine similarity checks to detect data exfiltration (leaking the system prompt).
  • Role/Token Detection: Identifying jailbreak tokens (e.g., "DAN").
  • Harmful Content: Using a classifier (Llama Guard 2) to detect unsafe outputs.
• Semantic Drift Detection: Measures the cosine distance between the live response and a reference persona-compliant response to detect behavioral manipulation.
• Escalation: Responses triggering high-risk predicates are blocked or escalated for human review.

Continuous Audit Loop

A cross-cutting component that aggregates structured logs from all layers. It supports:

• Alerting: Detecting sustained adversarial probing or coordinated campaigns.
• Retraining: Feeding confirmed true positives and false positives back into the Layer 1 semantic classifier to adapt to emerging attack patterns.

3. Key Contributions

1. Formal Threat Model: Defines a black-box adversary model for RAG chatbots, covering direct and indirect injection vectors and three specific attacker objectives: Instruction Override (IO), Data Exfiltration (DE), and Behavioral Manipulation (BM).
2. Three-Layer Defense Architecture: A coordinated framework that combines input screening, privilege-constrained context assembly, and output auditing. It is the first to evaluate the combined and individual contributions of these mechanisms against both direct and indirect injection.
3. Controlled Evaluation: A comprehensive evaluation on 5,080 samples (benign and adversarial) across three diverse LLMs (GPT-4o, Llama 3 8B, Mistral 7B).
4. Ablation and Error Analysis: Quantifies the marginal contribution of each layer and identifies residual bypass mechanisms, providing a roadmap for future improvements.

4. Experimental Results

The framework was evaluated against an undefended baseline, single-layer baselines, and a published guardrail system (NeMo Guardrails).

• Attack Success Rate (ASR) Reduction: The full three-layer framework reduced the macro-averaged ASR from 71.4% (undefended) to 11.3%.
  • This represents a 60.1 percentage-point reduction from the baseline.
  • It outperformed the best single-layer baseline (Layer 3 only, 38.6%) by 27.3 percentage points.
  • It outperformed NeMo Guardrails (35.1%) by 23.8 percentage points.
• False Positive Rate (FPR): The framework maintained an FPR of 4.8%, which is lower than NeMo Guardrails (8.4%) and comparable to Layer 1 alone (5.1%).
• Latency Overhead: The median end-to-end latency overhead was 61.2 ms, dominated by semantic encoding operations. This is within the operational budget for interactive chatbots.
• Ablation Insights:
  • The combined effect of the three layers exceeded the sum of individual contributions.
  • Layer 2 (Privilege Constrained) was found to suppress false positives in downstream layers (Layer 3) by filtering injections before they reach the output auditor.
  • No subset of two layers provided coverage equivalent to the full framework.

5. Significance and Claims

The paper claims that prompt injection is a structural vulnerability that cannot be solved by model upgrades alone; stronger models (e.g., GPT-4o) showed only marginally better resistance than smaller models when undefended, as instruction-following capability correlates with susceptibility to injected instructions.

The primary significance of the work is the demonstration that defense must be distributed across the entire inference pipeline. The authors argue that:

• Layering is necessary, not just beneficial: Direct and indirect injections enter at different points, requiring defenses positioned at those specific entry points.
• Model-agnosticism is critical: The framework functions without modifying LLM weights, making it deployable across diverse model families.
• Complementary Protection: The interaction between layers (e.g., Layer 2 reducing the load on Layer 3) creates a security posture that is greater than the sum of its parts.

The authors acknowledge limitations, including the static black-box attacker model, the focus on single-turn queries (multi-turn attacks remain a residual risk), and the sensitivity of thresholds to distribution shifts. They position the framework as a practical foundation for securing RAG systems, with future work needed to address multi-turn accumulation and implicit instructions in retrieved documents.