💻 computer science

Can It Reach the Generator? Investigating the Survival of Prompt-Injection Attacks in Realistic RAG Settings

This paper demonstrates that in realistic RAG systems, most prompt-injection attacks fail to reach the generator due to retrieval and reranking filters, and those that do survive are easily detected by lightweight guard models, significantly overestimating the actual security risks of prior studies.

Original authors: Yu Yin, Shuai Wang, Bevan Koopman, Guido Zuccon

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yu Yin, Shuai Wang, Bevan Koopman, Guido Zuccon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a very smart, high-tech library. When a visitor asks, "What's the best coffee maker?", your library doesn't just hand them a list of books. Instead, it has a three-step team that decides what to show them:

The Librarian (Retriever): Scans the entire library to find the top 10 books that might be relevant.
The Expert Critic (Reranker): Reads those top 10 books carefully and re-orders them to make sure the absolute best ones are at the very top.
The Concierge (Generator): Takes the top 5 books from the Critic and writes a final recommendation letter for the visitor.

The Problem: The "Fake Book" Attack
Recently, researchers discovered a trick called "Generative Engine Optimization" (GEO). It's like someone trying to sneak a fake, self-promoting book into the library to trick the Concierge into recommending it as the best coffee maker, even if it's terrible.

Previous studies claimed this trick was a massive success, saying attackers could get their fake book to the top of the list 80% of the time. They made it sound like the library was wide open to sabotage.

The Reality Check: The Paper's Discovery
This paper says, "Wait a minute. Those previous studies were cheating."

In those old studies, they assumed the fake book was already sitting on the Critic's desk, skipping the Librarian and the Critic entirely. They just asked the Concierge, "Here is a fake book; can you recommend it?"

But in the real world, the fake book has to survive the Librarian and the Critic first. The authors tested this realistic scenario and found the attack is much weaker than people thought.

Here is what they found, using simple analogies:

1. The Librarian Filters Out the Fakes

When an attacker tries to write a fake book to trick the system, they often have to write it in a weird, robotic way (like adding secret codes or weird instructions).

The Result: The Librarian, who is looking for normal-sounding books, often doesn't even pick up the fake book. It gets left on the shelf.
The Stat: About 20% of the fake books never even make it past the Librarian to the Critic.

2. The Critic Rejects the Weird Stuff

Even if the fake book gets past the Librarian, it has to face the Expert Critic. The Critic reads the content to see if it actually answers the question.

The Result: Many attacks that looked strong in the old "cheated" tests fall apart here. The Critic realizes the book is trying too hard to trick them and pushes it down the list.
The Stat: Attacks that rely on "gradient-based" math (complex, robotic code) almost completely fail. Their success rate drops from looking like a threat to less than 2%. They simply can't survive the Critic.

3. Only the "Smooth Talkers" Survive

There is one type of attack that still works: LLM-driven attacks. These are fake books written by another AI that sounds very natural and persuasive.

The Result: These "smooth talkers" can trick the Librarian and the Critic because they sound like real, helpful recommendations.
The Stat: These attacks still work about 40–50% of the time. They are the only ones that make it all the way to the Concierge.

4. The "Smoke Detector" Works

The paper also tested if we can catch these attackers.

The Discovery: They built a tiny, lightweight "security guard" (a small AI model) trained on just a few examples of these attacks.
The Result: This guard caught every single attack that made it through the pipeline, including the smooth-talking ones. It's like having a smoke detector that instantly smells the smoke, no matter how well the fire was hidden.

The Big Takeaway

The paper concludes that the "apocalypse" scenario where AI recommendations are easily hijacked is exaggerated.

Old View: "Hackers can easily force AI to recommend anything!" (Based on tests that skipped the security steps).
New View: "Hackers can try, but the Library's natural security steps (the Librarian and Critic) filter out most of them. The few that get through are easy to spot with a simple detector."

In short, the system is much more robust than we feared, but we still need to keep our "smoke detectors" (security guards) turned on.

Technical Summary: Can It Reach the Generator? Investigating the Survival of Prompt-Injection Attacks in Realistic RAG Settings

1. Problem Formulation

Recent research in Generative Engine Optimization (GEO) has demonstrated that prompt-injection attacks can manipulate Large Language Model (LLM) outputs to prioritize specific products, with reported success rates as high as 80%. However, the authors argue that these prior evaluations rely on a "Frozen Context" (FC) protocol that is fundamentally unrealistic for deployed systems. In the FC protocol, the attacked document is assumed to be automatically retrieved and passed directly to the generator, bypassing the retrieval and reranking stages entirely.

In real-world Retrieval-Augmented Generation (RAG) pipelines, an adversarial edit to a document's content alters its semantic and lexical properties. This modification can cause the document to fail retrieval by the initial retriever or be filtered out by a reranker before it ever reaches the generator. The central research question is whether GEO attacks can survive the full three-stage pipeline (Retriever $\to$ Reranker $\to$ Generator) or if their effectiveness is an artifact of simplified evaluation protocols.

2. Methodology

The authors propose and evaluate a realistic, end-to-end three-stage RAG pipeline:

Retriever: A sparse (BM25) or dense (BAAI/bge-large-en-v1.5) retriever that ranks the entire corpus.
Reranker: An LLM-based listwise reranker (Qwen3-8B using RankGPT strategy) that re-orders the top- $n$ (10) retrieved candidates.
Generator: An LLM (Qwen3-8B) that generates a response based on the top- $k$ (5) reranked candidates, producing $m$ (3) product recommendations.

Experimental Setup

Dataset: Amazon ESCI product-search corpus (US locale), filtered to 1,294 queries with $\ge$ 40 annotated products to reflect e-commerce scale.
Attack Targets: Documents at Rank 6 (first to be dropped by the reranker) and Rank 10 (last seen by the reranker) of the unattacked baseline.
Attack Methods: Seven attacks from three categories were evaluated:
- Instruction Override (IOA): Fixed suffixes (e.g., DCH schema).
- LLM-driven: Iterative optimization using an auxiliary LLM (CORE-Review, CORE-Reason, TAP).
- Gradient-based: White-box token optimization (RAF, SRP, STS).
Evaluation Protocols:
- Frozen Context (FC): Prior work's method (bypasses retrieval/reranking).
- End-to-End (E2E): The attacked corpus is re-indexed, and the full pipeline is executed.
Metrics:
- Retrieval Survival ( $S_r$ ): Does the attacked document enter the reranker's top-10?
- Reranking Exposure ( $E_\rho$ ): Does it enter the generator's top-5 input?
- Generation Success ( $S_g$ ): Does the target product appear in the final top-3 recommendation?
Defense Evaluation: Tested off-the-shelf guards (Llama-Guard, QwenGuard, Prompt-Guard) and a lightweight Prompt-Guard finetuned on a small dataset (104 queries).

3. Key Findings and Results

3.1 Prior Protocols Overstate Effectiveness

The Frozen Context protocol significantly inflates attack success. When the full pipeline is considered:

Retrieval acts as a filter: On average, ~20% of attack documents fail to be retrieved at all. Instruction Override (IOA) attacks suffer the most, with retrieval survival dropping to ~59% because their injected suffixes dilute query-relevant content.
Reranking reshapes outcomes: The reranker further filters or promotes documents. While it can boost some attacks by ~16.5% via rank promotion, the subsequent truncation to the top-5 input for the generator mitigates many attacks.

3.2 End-to-End Attack Effectiveness is Highly Variable

Under the E2E pipeline, attack performance collapses for most methods:

Gradient-based attacks (RAF, SRP, STS): These fail completely, with success rates ( $S_g$ ) dropping below 2%. They are optimized for the generator objective but degrade the document's lexical/semantic match for the retriever and reranker.
Instruction Override (IOA): Success drops from high FC rates to ~14.7% E2E due to retrieval failure.
LLM-driven attacks (CORE-Review, CORE-Reason, TAP): These are the only methods remaining effective end-to-end.
- CORE-Reason is unique: It actually sees a ~4% increase in success under E2E compared to FC. Its optimization strategy includes the user query multiple times, which improves retrieval survival and is actively promoted by the dense retriever.
- CORE-Review and TAP maintain success rates of ~46% and ~40% respectively, but still suffer significant drops compared to FC.

3.3 Position Bias and Stage Preferences

The study reveals opposing position biases between stages:

Reranker: Prefers attacks placed later in the input (Rank 10 > Rank 6).
Generator: Prefers attacks placed earlier in the input (Rank 6 > Rank 10), consistent with "lost-in-the-middle" phenomena.
This suggests that a single-stage analysis misses critical dynamics where an attack might succeed at one stage but fail at another due to input position.

3.4 Defensibility

Off-the-shelf guards (Llama-Guard, QwenGuard) fail to generalize, often having high False Discovery Rates (FDR) or missing sophisticated LLM-driven attacks (CORE-Review/Reason).
Lightweight Finetuning: A Prompt-Guard model finetuned on only 104 training queries (1:1 balanced attack/clean) achieved near-perfect detection (97.3% F1) across all attack types, including the most effective end-to-end ones. This indicates that surviving attacks expose a learnable schema.

4. Significance and Claims

The paper claims that the perceived threat landscape of GEO prompt-injection attacks is narrower than previously reported.

Revised Threat Model: Most existing attacks (gradient-based and instruction override) are ineffective in realistic RAG settings because they cannot survive the retrieval and reranking stages.
Necessity of E2E Evaluation: The authors argue that future GEO research must adopt end-to-end evaluation protocols. Relying on frozen-context assumptions leads to false alarms regarding system robustness.
Detectability: Even the attacks that do survive the full pipeline (LLM-driven prompt optimizations) follow optimization schemas that are easily detectable with minimal data, suggesting that robust defenses are feasible without compromising system utility.

The study concludes that while GEO attacks remain a concern, the specific vulnerabilities are highly dependent on the attack method and the pipeline architecture, and the "survival" of an attack through the full RAG pipeline is the critical metric for assessing real-world risk.