🤖 machine learning

Semantic Chameleon: Corpus-Dependent Poisoning Attacks and Defenses in RAG Systems

This paper introduces "Semantic Chameleon," a gradient-guided corpus poisoning attack that manipulates RAG system outputs by optimizing dual documents to hijack retrieval, demonstrating that while hybrid retrieval effectively mitigates attacks on vector-only systems, jointly optimized payloads can still partially bypass these defenses, with success rates varying significantly across different LLM families and datasets.

Original authors: Scott Thornton

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Scott Thornton

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, helpful assistant (an AI) who is great at answering questions but doesn't know everything about the world. To fix this, you give them a giant library of books (a "corpus") to look up information before they answer. This setup is called RAG (Retrieval-Augmented Generation).

The paper "Semantic Chameleon" is a security report about how bad guys can sneak into that library, hide fake books, and trick the assistant into giving dangerous answers. But more importantly, it discovers that the type of library you have matters just as much as the security guard you hire.

Here is the breakdown in simple terms:

1. The Attack: The "Trojan Horse" Books

The researchers found a way for hackers to inject two specific "poisoned" books into the library:

The Sleeper: A book that looks totally normal and boring. It talks about legitimate topics but has hidden "hooks" (like specific phrases) that make it look similar to the hacker's target in the computer's mind.
The Trigger: A book that looks mostly normal at the start but contains the actual malicious instructions (like "how to bypass security") hidden inside.

How it works: When a user asks a specific question, the AI's search engine finds the "Sleeper" first because it looks similar. The AI then reads the "Trigger" attached to it and accidentally follows the bad instructions.

2. The Big Discovery: The Library Matters (The "Chameleon" Effect)

The paper's biggest surprise is that where you put these fake books changes everything. They tested this in two different "libraries":

The General Library (Wikipedia): Imagine a library with books about everything—cooking, history, space.
- Result: If a hacker tries to sneak in a book about "hacking," it stands out like a neon sign. The search engine immediately flags it as weird because the word "hack" rarely appears in a cooking book. The attack fails because the fake book gets caught immediately.
The Technical Library (Security Stack Exchange): Imagine a library full of complex computer manuals and security docs.
- Result: Here, words like "bypass," "injection," and "exploit" are common. The fake book blends in perfectly, like a chameleon changing color to match its leaves. The search engine can't tell the difference between a real security manual and the fake one. The attack works much better here.

The Lesson: A security defense that works on a general library might fail completely on a technical one. You can't use a "one-size-fits-all" security rule.

3. The Solution: The "Double-Check" System (Hybrid Retrieval)

The researchers tested a simple fix: Hybrid Retrieval.

Think of the search engine as a librarian.

Old Way (Vector Search): The librarian only looks at the vibe or meaning of the words. "Does this book feel like it belongs?" If the hacker mimics the vibe perfectly, the librarian lets it in.
New Way (Hybrid Search): The librarian now uses two methods:
1. The Vibe Check: Does it feel right? (Vector search)
2. The Keyword Check: Does it actually contain the specific words the user asked for? (BM25 search)

The Result: The hackers were great at mimicking the "vibe," but they couldn't perfectly match the specific keywords without making the book look suspicious. By forcing the librarian to use both checks, the fake books were rejected 100% of the time against standard attacks.

4. The Smart Hacker vs. The Double-Check

The researchers asked: "What if the hacker is really smart and tries to game both systems?"

They tried to write books that sounded right and had the right keywords.
Outcome: They could get the books into the library about 20–44% of the time. It wasn't a perfect defense, but it made the job much harder for the hacker. It raised the "attack bar" significantly.

5. The AI's Personality Matters Too

They tested this against five different AI models (like different personalities of assistants):

The "Strict" AI (Claude): Even if it found the bad book, it refused to follow the bad instructions. It was like a guard dog that barks at the intruder but doesn't attack the owner.
The "Loose" AI (Llama): This one was very easy to trick. If it found the bad book, it happily followed the instructions.
The "New" AI (GPT-5.3): Better than the old ones, but still got tricked nearly half the time.

The Takeaway: You can't just rely on the AI being "safe." You need to secure the library (the search) and train the AI.

Summary for the Everyday Person

Hackers can hide bad instructions inside normal-looking documents in your AI's knowledge base.
Context is King: If your AI works with technical jargon, it's harder to detect these hacks because the bad words look normal. If it works with general topics, the bad words stand out.
The Best Defense: Don't just rely on one type of search. Use a hybrid system that checks both the meaning and the exact words. This simple change stopped almost all standard attacks in the study.
Layered Security: Even with a good search system, you need to check which AI model you are using, as some are much easier to trick than others.

In short: To keep your AI safe, you need a smart librarian who checks both the "vibe" and the "keywords" of the books, and you need to know that some libraries are harder to protect than others.

1. Problem Statement

Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLMs) with external knowledge but introduce a critical attack surface: the retrieval mechanism. Adversaries can inject poisoned documents into the knowledge base to manipulate model outputs without altering model weights or system prompts.

The paper identifies a gap in current research: while poisoning attacks are known to be feasible, the corpus composition (the domain and vocabulary of the knowledge base) is rarely treated as a primary security variable. The authors hypothesize that the effectiveness of both attacks and defenses varies significantly depending on whether the corpus is technical (specialized vocabulary) or general (encyclopedic).

2. Methodology

Attack Design: Dual-Document Poisoning

The authors employ a dual-document strategy optimized via Greedy Coordinate Gradient (GCG) to bypass retrieval filters:

Sleeper Document: Contains mostly benign content with specific "semantic hooks" to bridge the gap between benign concepts and the attack domain, ensuring stealth (low retrieval on benign queries).
Trigger Document: Contains the malicious payload, preceded by a benign prefix to maintain semantic coherence with the sleeper.
Optimization: The attacker maximizes cosine similarity in the embedding space for target queries while minimizing similarity for benign queries.

Experimental Setup

Corpora:
- Security Stack Exchange (Technical): 67,941 documents with specialized security vocabulary.
- FEVER Wikipedia (General): 96,561 encyclopedic articles with broad vocabulary.
- Production Case Study: 156,777 documents from a major network security vendor.
Retrieval Architectures:
- Pure Vector: Dense retrieval only.
- Hybrid Retrieval: A weighted combination of BM25 (sparse/keyword) and Vector (dense) scores: $score = \alpha \cdot \hat{v} + (1-\alpha) \cdot \hat{b}$ .
LLMs Evaluated: Five models across four providers (GPT-5.3, GPT-4o, GPT-4o-mini, Claude Sonnet 4.6, Llama 4).
Metrics: Co-retrieval (both documents retrieved), Stealth (low retrieval on benign queries), and Overall Success (co-retrieval + stealth).

3. Key Contributions

Hybrid Retrieval as an Architectural Defense: The paper demonstrates that a simple hybrid retriever (BM25 + Vector) is a highly effective first-line defense against gradient-guided poisoning, reducing attack success from 38% to 0% in many scenarios.
Corpus-Dependent Security: The study establishes that corpus domain dictates the attack-defense balance. Technical corpora allow for high stealth but low retrieval success, while general corpora allow high retrieval but zero stealth due to vocabulary anomalies.
Multi-Model Resilience Analysis: The authors evaluate the full attack chain across five LLMs, revealing that safety training maturity varies drastically between models, with open-weights models (Llama 4) being significantly more vulnerable than proprietary ones.
Production-Scale Validation: A preliminary case study on a real-world vendor corpus confirms that attacks optimized for one corpus do not transfer to another without adaptation.

4. Key Results

A. Corpus-Dependent Attack Effectiveness

Security Stack Exchange (Technical): Achieved 66.7% stealth but only 11.1% overall success. The specialized vocabulary "absorbs" attack terms, making them less anomalous, but the semantic specificity makes co-retrieval difficult.
FEVER Wikipedia (General): Achieved 100% co-retrieval but 0% stealth. Attack terms (e.g., "bypass," "injection") are highly anomalous in general text, causing the documents to be retrieved indiscriminately on benign queries.
Detection Gap: Detection difficulty varies 13–62× between corpora. Keyword-based detection fails on technical corpora but works perfectly on general ones.

B. Hybrid Retrieval Defense Performance

Gradient-Only Attacks: Hybrid retrieval (using $\alpha \le 0.5$ ) reduced gradient-guided attack success from 38% (Pure Vector) to 0% across all tested mixing weights. The BM25 component effectively filters out semantically similar but keyword-poor poisoned documents.
Joint Optimization (Advanced Attack): When attackers optimize for both sparse (BM25) and dense (Vector) channels simultaneously, hybrid retrieval is partially circumvented, achieving 20–44% success. However, this remains significantly lower than the 38% success rate on pure vector systems.
Keyword Injection: Attempts to inject keywords to satisfy BM25 failed to maintain stealth, as high keyword density triggered anomaly detection.

C. Multi-Model End-to-End Evaluation

Attack success rates varied dramatically by model, even when poisoned documents were successfully retrieved:

Llama 4 Instruct: 93.3% attack success, 93.3% safety violations. (Most vulnerable).
GPT-5.3: 46.7% attack success, 33.3% safety violations. (Most resilient).
Claude Sonnet 4.6: 60.0% attack success, but only 6.7% safety violations. (Strong safety training prevents actionable output despite retrieval).
Conclusion: Model-level safety is necessary but insufficient; retrieval-level defenses are critical.

D. Detection Signals

Query Pattern Differential (QPD): The most robust detection method across corpora. It measures the ratio of a document's retrieval frequency on sensitive vs. benign queries. It showed the smallest corpus gap (3.7×) compared to static methods like keyword anomaly (40× gap).

5. Significance and Recommendations

Architectural Shift: The paper argues that security in RAG should not rely solely on LLM safety training or post-retrieval detection. Instead, the retrieval architecture itself must be hardened.
Immediate Defense: The authors recommend deploying Hybrid Retrieval with $\alpha \le 0.5$ (prioritizing BM25 or balanced weighting) as an immediately deployable first-line defense. This eliminates the most common gradient-based attacks.
Layered Approach: For sophisticated attackers using joint optimization, hybrid retrieval should be supplemented with QPD monitoring and rigorous model-level safety evaluation.
Benchmarking Implications: Future RAG security benchmarks must systematically vary corpus domains. A defense that works on general knowledge (Wikipedia) may fail on technical documentation, and vice versa.

In summary, "Semantic Chameleon" provides empirical evidence that RAG security is not a monolithic problem but is deeply dependent on the interaction between the knowledge base's vocabulary, the retrieval architecture, and the target LLM's safety training.