💻 computer science

"Someone Hid It": Query-Agnostic Black-Box Attacks on LLM-Based Retrieval

This paper proposes a practical, query-agnostic black-box attack method that generates transferable adversarial tokens using zero-shot surrogate LLMs to manipulate LLM-based retrieval systems, revealing significant robustness risks even without access to victim models or specific queries.

Original authors: Jiate Li, Defu Cao, Li Li, Wei Yang, Yuehan Qin, Chenxiao Yu, Tiannuo Yang, Ryan A. Rossi, Yan Liu, Xiyang Hu, Yue Zhao

Published 2026-05-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiate Li, Defu Cao, Li Li, Wei Yang, Yuehan Qin, Chenxiao Yu, Tiannuo Yang, Ryan A. Rossi, Yan Liu, Xiyang Hu, Yue Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Hiding a Book in a Library

Imagine a massive, high-tech library where a robot librarian (the LLM-based Retrieval System) helps you find books based on what you ask. If you ask for "recipes," the robot pulls out the cookbooks. If you ask for "history," it pulls out history books.

The researchers in this paper discovered a way to trick the robot librarian into ignoring a specific book entirely, even if you ask for the exact topic that book covers. They call this a "Query-Agnostic Black-Box Attack."

Let's break down what that means and how they did it.

1. The Problem with Previous "Hacks"

Before this study, people who tried to mess with these library robots had two big problems:

They needed the keys: Most previous hacks required knowing exactly how the robot was built (its internal code and brain). In the real world, you can't see inside a company's secret robot.
They needed to know your question: Most hacks only worked if the attacker knew exactly what question you were going to ask before they messed with the book. But in reality, the attacker doesn't know what you will ask until after they've already edited the book.

The Paper's Goal: Create a "universal invisibility cloak" for a document. The attacker wants to edit a document (like a Wikipedia page or a Reddit comment) just enough so that no matter what question a user asks later, the robot librarian will fail to find it. And they want to do this without seeing the robot's brain.

2. The Solution: The "Surrogate" Robot

Since the attacker can't see the victim's robot, they build their own "practice robot" (called a Surrogate Model) to test their tricks on.

The Analogy:
Imagine you are trying to sneak a book past a specific security guard, but you can't see the guard. So, you hire a look-alike guard (the Surrogate) and practice your sneaking techniques on them. If you can fool the look-alike, you hope it will also fool the real guard.

3. The Secret Sauce: "Topic Clusters"

The researchers noticed something interesting about how these robots think. They found that the robot groups similar topics together in its "mind."

If you show the robot 10 different articles about Science, the robot sees them as a tight cluster of friends.
If you show it an article about Movies, it sees that as a different group of friends.

The Attack Strategy:
The researchers realized that to hide a document, they don't need to make it look like a movie. They just need to push it out of its Science cluster and make it look like it belongs to a different group (or nowhere at all).

4. How the Attack Works: The "Adversarial Dance"

To achieve this, the researchers used a clever two-step dance called Query-Document Adversarial Learning:

The Fake Questions: They use a third-party AI to generate many fake questions that a user might ask about the document (e.g., "Tell me about economics," "What is the history of money?").
The Push and Pull:
- Step A (The Push): They tweak the document slightly (adding invisible "noise" words) to make it look bad to the fake questions.
- Step B (The Pull): They tweak the fake questions to make them look very similar to the document.
- The Loop: They repeat this dance over and over. The document gets pushed further and further away from the "Science" group, while the fake questions pull it toward a "confused" area.

The Result: The document gets a few invisible "glitch words" added to the end of it. To a human, it looks fine. But to the robot librarian, the document now looks like it belongs to a completely different topic, so it gets ignored.

5. Why It's Scary (and Important)

It Works Everywhere: The researchers tested this on many different types of robot librarians (Qwen, Gemma, Jina, etc.). Once they added the "glitch words" to a document, all of them failed to find it. It's like a universal key that locks every door.
It's Zero-Shot: They didn't need to know the specific robot they were attacking. They just trained on their "practice robot," and it worked on the real one.
It's Realistic: The attacker only needs to be able to edit a document slightly (like adding a comment or a small edit), just like a normal user on a website. They don't need to be a hacker with superpowers.

6. The "Invisible Ink" Discovery

The researchers found a surprising trick to make the attack even better. Instead of looking at the robot's "final answer" (the deep thinking), they looked at the robot's "first impression" (the initial word processing).

Analogy: It's like realizing that if you change the font color of a book cover slightly, the librarian notices it immediately, whereas changing the last sentence of the book (the "final answer") doesn't matter as much. Using this "first impression" layer made their attack much stronger.

7. What This Means for the Future

The paper concludes that these retrieval systems are more fragile than we thought.

Benign Accidents: Even if no one is trying to hack, a normal user accidentally editing a document (like fixing a typo or adding a reply) could accidentally trigger this "hiding" effect, causing the document to vanish from search results.
No Defense Yet: The paper notes that current security measures (like checking for weird text) don't stop this specific type of attack.

In Summary: The researchers showed that by adding a few invisible, carefully chosen words to a document, you can trick smart AI search engines into "forgetting" that document exists, even if you don't know how the search engine works or what people will ask about it later. It's a "ghosting" technique for the digital age.

Technical Summary: "Someone Hid It!" Query-Agnostic Black-Box Attacks on LLM-Based Retrieval

1. Problem Formulation

The paper addresses the security vulnerabilities of Large Language Model (LLM)-based Retrieval (LLMR) systems, which serve as backbones for Retrieval-Augmented Generation (RAG), Information Retrieval (IR), and Agent Memory. While existing research has demonstrated that LLMR is susceptible to adversarial attacks via token-level injections, current methods suffer from two critical limitations regarding real-world applicability:

White-Box Assumptions: Most existing attacks require full access to the victim model's parameters or architecture.
Query Knowledge: Existing methods often presume the attacker knows the specific user query or has access to the victim's document corpus.

In practical black-box scenarios, attackers typically lack knowledge of the victim model, the hidden document corpus, and the specific user queries (which are generated post-attack). Furthermore, attackers usually have limited capabilities, such as a normal user (e.g., a Wikipedia contributor) who can only read and slightly edit a document.

The Core Problem: Can an attacker successfully hide a specific victim document from an LLM-based retriever in a complete black-box setting? The attacker must manipulate the document (via token injection) without knowing the victim model, the user query, or the full corpus, such that the document fails to appear in the top- $k$ results for any relevant query.

2. Methodology

The authors propose a Query-Agnostic Black-Box Attack method that generates transferable injection tokens using a surrogate LLM. The methodology is built upon a theoretical framework and an adversarial learning mechanism.

2.1 Theoretical Framework: $\epsilon$ - $p_\epsilon$ -Precise Retriever

The authors first establish a theoretical definition to quantify the behavior of LLM retrievers. They define an $\epsilon$ - $p_\epsilon$ -Precise Retriever as a system where, for a given topic distribution, a fraction $p_\epsilon$ of contexts within the same topic have embedding similarities that are at least $\epsilon$ higher than similarities between contexts of different topics.

Empirical Verification: Experiments using Principal Component Analysis (PCA) on embeddings of 24 topics confirm that LLMs naturally cluster contexts of the same topic in the embedding space, validating the assumption that retrievers exhibit this property.

2.2 Transferability via Min-Max Optimization

To achieve a black-box attack, the method utilizes a surrogate model ( $g$ ) to approximate the victim model ( $f$ ). The attack is formulated as a min-max problem:

Objective: Minimize the similarity between the perturbed document $d'$ and the query $q$ .
Challenge: Since $q$ is unknown, the authors approximate the target by maximizing the similarity between $d'$ and a set of sampled queries $X$ within the same topic.
Theorem 3.2 (Transferability): The authors prove that if the surrogate model $g$ is $\epsilon_g$ - $p_{g\epsilon}$ -Precise, and the attack optimizes $d'$ such that its similarity to sampled queries is lower than the minimum similarity within the topic cluster, then $d'$ will likely be pushed out of the top- $k$ results for the victim model $f$ (assuming $f$ is also precise).

2.3 Document-Query Adversarial (DQ-A) Learning

To solve the min-max problem without knowing the true query, the authors propose a Document-Query Adversarial (DQ-A) learning framework:

Query Sampling: A third-party "casual" LLM generates $N$ diverse potential queries ( $S$ ) related to the victim document's topic.
Adversarial Optimization:
- Document Optimization: The injected tokens in the victim document are optimized to minimize similarity with the sampled queries.
- Query Optimization: The sampled queries are optimized to maximize similarity with the victim document (acting as a "hard negative" generator).
Population Difference: Instead of using learning rates to balance the inner (query) and outer (document) loops, the method uses a population difference. It optimizes a set of $N$ diverse queries against a single document, simulating the effect of a larger learning rate for the query side.

2.4 Word-Embedding Surrogate Strategy

A key technical innovation is the use of word-embeddings (the output of the first embedding layer) rather than the last-hidden states (final transformer block) as the surrogate for adversarial learning.

Rationale: Empirical findings suggest that optimizing distances in the shallow layers (word-embeddings) yields more effective and transferable attacks on the final retrieval embeddings than optimizing the deep layers directly. This is attributed to the high parameter count of LLMs, where shallow-layer perturbations have a more robust influence on the final representation space.

3. Key Contributions

Black-Box Attack Framework: The paper presents the first complete black-box attack method for LLMR that requires no knowledge of the victim model, the document corpus, or the user queries. It simulates a realistic attacker capable only of editing documents.
Theoretical Foundation: It establishes a theoretical framework ( $\epsilon$ - $p_\epsilon$ -Precise Retriever) and provides empirical evidence that LLM embeddings cluster by topic, which is the basis for proving attack transferability.
DQ-A Learning Mechanism: The authors introduce a novel adversarial learning mechanism that jointly optimizes victim documents and sampled queries using a population-based approach to handle non-convex optimization challenges.
Surrogate Strategy: The paper identifies and validates that using word-embeddings as a surrogate for adversarial training significantly improves attack transferability compared to using last-hidden states.
Zero-Shot Transferability: The method demonstrates that tokens injected into a document can simultaneously degrade retrieval performance across various different LLM retrievers (e.g., Qwen, Mistral, Jina, Gemma) without retraining or model-specific tuning.

4. Experimental Results

The method was evaluated on benchmark datasets (Economics, Psychology, Biology, Robotics) from the BRIGHT benchmark against seven popular LLM retrievers (including Qwen-1.5, SFR-Mistral, E5-Mistral, Embedding-Gemma, Jina, Granite, and Qwen3).

Performance Impact: The proposed attack (DQ-A) consistently outperformed existing baselines (GCG, PRADA, Poisoned-RAG).
- It achieved an average performance drop of 2%–10% in Recall@25 and 1%–7% in Recall@50 across most models.
- In specific cases (e.g., Robotics dataset on Qwen1.5), the attack caused an 8% drop in Recall@50, representing a ~30% reduction in the model's original performance.
Ablation Studies:
- Adversarial Learning: Removing the adversarial query optimization (S-GCG) resulted in significantly lower attack success, validating the DQ-A design.
- Surrogate Layer: Using word-embeddings (DQ-A) outperformed using last-hidden states (DQ-A-H) in most scenarios.
- Sampling Size: Increasing the number of sampled queries ( $|S|$ ) improved performance up to a point (around 10), after which diminishing returns were observed.
- Injection Position: Attacks were most effective when tokens were injected at the beginning or end of the document, with the middle being less effective, aligning with transformer attention mechanisms.
Robustness: The attack remained effective even when the retriever was paired with a reranker model, as the attack filters the document out of the candidate pool before the reranker can act.

5. Significance and Claims

The paper claims that its work reveals a critical, previously underexplored failure mode in LLM-based retrieval systems.

Real-World Relevance: By operating under strict black-box constraints (no query knowledge, no model access), the attack highlights a vulnerability that exists in practical deployment scenarios, such as public wikis or discussion forums where content can be edited by untrusted users.
Broader Implications: The authors note that the robustness issues identified may not only stem from malicious attacks but could also arise from benign or unintended document edits in the real world. If standard content updates can inadvertently push documents out of retrieval clusters, the reliability of LLMR systems is fundamentally challenged.
Defense Gap: The paper explicitly states that existing defense mechanisms (focused on corpus poisoning detection or post-retrieval reranking) are insufficient for this type of "decreasing" attack. It calls for new defense strategies that focus on building robust retrieval mechanisms themselves, rather than just detecting poisoned outputs.

The authors maintain a modest tone, acknowledging that while the attack is effective, it is intended to characterize system vulnerabilities to inform the development of more robust systems, rather than to promote adversarial misuse. They also note that boosting specific documents using a reversed version of the attack is possible but currently limited in effectiveness.