Highlight & Summarize: RAG without the jailbreaks

1. Problem Statement

Retrieval-Augmented Generation (RAG) systems are widely used to ground Large Language Models (LLMs) in trusted knowledge bases (e.g., corporate FAQs, policy documents). However, these systems remain vulnerable to jailbreaking and model hijacking.

• The Threat: Malicious users can craft adversarial prompts (inputting specially designed text) to bypass safety filters, force the LLM to generate offensive content, misrepresent company policy, or repurpose the model for unintended tasks (e.g., generating spam or legal commitments).
• Limitations of Current Defenses: Existing mitigation strategies (hardening system prompts, input classifiers, or fine-tuning) are probabilistic. They are often bypassed due to the vast space of possible inputs and the difficulty of anticipating every adversarial variation.

2. Methodology: Highlight & Summarize (H&S)

The authors propose Highlight & Summarize (H&S), a new design pattern that mitigates these attacks by design rather than by detection. The core principle is to ensure the generative LLM (the summarizer) never receives the user's raw input.

The Architecture

The H&S pipeline splits the standard RAG process into two distinct, isolated components:

1. The Highlighter (Retrieval & Extraction):
   • Takes the user's question and the retrieved documents.
   • Extracts ("highlights") relevant text passages from the trusted documents.
   • Security Constraint: The output must be verbatim, contiguous, non-overlapping substrings of the trusted documents. It cannot generate new text or modify the source.
   • Implementation Variants: The paper tests five highlighter types, including zero-shot LLMs (with fuzzy matching), structured LLMs, Chain-of-Thought (Two Steps), span-only LLMs, and fine-tuned extractive QA models (DeBERTaV3).
2. The Summarizer (Generation):
   • Receives only the highlighted text passages.
   • Crucially: It does not see the user's original question.
   • It infers the intent of the highlighted text and synthesizes a coherent answer.
   • The user's question is effectively "burned" after the highlighting step.

Security Mechanism

• Input Isolation: Since the summarizer LLM only processes trusted text extracts, an attacker cannot inject adversarial prompts directly into the generative model.
• Min-Words Constraint: The system enforces a security parameter min_words. Any extracted passage must be a contiguous match of at least min_words from the source. This prevents attackers from "chaining" together small fragments of text to form a malicious prompt (similar to Return-Oriented Programming in binary exploitation).

3. Key Contributions

1. Novel Design Pattern: Introduction of H&S as a system-level defense that deterministically limits attacker control over the generative LLM.
2. Theoretical Formalization:
   • Defined the "Control Region" of an LLM (the set of outputs an attacker can reliably induce).
   • Proved (via Lean formalization) that H&S exponentially decreases the size of the attacker's control region compared to standard RAG, reducing the input space from arbitrary tokens to a subset of trusted contiguous strings.
3. Comprehensive Evaluation:
   • Evaluated against non-adaptive attacks (using the LLMail-Inject dataset).
   • Evaluated against adaptive attacks (where the attacker knows the defense and tries to manipulate the highlighter).
   • Compared performance against "Vanilla" RAG on two datasets: RepliQA (fictional policy documents) and BioASQ (biomedical questions).
4. Open Science: Release of code, experimental data, LLM-as-a-Judge ratings, and Lean 4 proofs.

4. Results

Security Performance

• Non-Adaptive Attacks: H&S achieved 0% success rate for jailbreaks on the LLMail-Inject dataset (1,028 prompts). In contrast, standard RAG had an 81% success rate for triggering prohibited tool calls.
• Adaptive Attacks: The min_words constraint effectively thwarted attempts to construct malicious sentences by piecing together document fragments.
• Highlighter Vulnerability: The study confirmed that the highlighter LLM itself is still vulnerable to jailbreaks, reinforcing the need for the system-level separation (the summarizer must remain isolated).

Quality and Accuracy

Contrary to the assumption that security comes at the cost of utility, H&S performed comparably or better than standard RAG:

• Elo Ratings: On the RepliQA dataset, the "Two Steps" H&S variant achieved an Elo score of 1170 vs. 1104 for Vanilla RAG. On BioASQ, H&S Structured scored 1181 vs. 1197 (comparable).
• Correctness: H&S variants (particularly LLM-based highlighters) often outperformed Vanilla RAG in correctness metrics.
• Hallucination & Faithfulness: While H&S had lower K-Precision (lexical overlap with reference answers) because it paraphrases rather than copies verbatim, it maintained high Recall and semantic correctness. The system forces the model to stick strictly to the provided context.
• Ablation: The study showed that the Summarizer is essential; using only the highlighter (extractive QA) resulted in significantly lower performance (e.g., 2% preference vs. 67% for full H&S on BioASQ).

5. Significance and Implications

• Paradigm Shift: Moves security from "detecting bad inputs" (probabilistic) to "preventing bad inputs from reaching the generator" (deterministic).
• Trustworthy AI: Enables the deployment of RAG systems in high-stakes environments (legal, corporate policy, customer support) where unintended outputs could have legal or reputational consequences.
• Scalability: The approach transforms the complex problem of scanning infinite user inputs for jailbreaks into the simpler, finite problem of scanning the trusted knowledge base for potential trigger strings.
• Future Directions: The authors note current limitations in handling complex reasoning (math, multi-step logic) and Yes/No questions, suggesting these as areas for future extension.

In conclusion, Highlight & Summarize demonstrates that it is possible to secure RAG systems against sophisticated adversarial attacks without sacrificing, and in some cases even improving, the quality and accuracy of the generated responses.