💻 computer science

PARASITE: Conditional System Prompt Poisoning to Hijack LLMs

The paper introduces PARASITE, a black-box attack framework that poisons third-party system prompts with "sleeper agents" to hijack Large Language Models into generating targeted, compromised responses for specific queries while maintaining normal functionality on benign inputs, effectively evading standard defenses.

Original authors: Viet Pham, Thai Le

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Viet Pham, Thai Le

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you buy a highly sophisticated, pre-made "instruction manual" for a super-smart robot assistant from a public online store. You download it because it looks helpful and promises to make the robot great at answering questions about history, math, and science.

This paper, titled PARASITE, reveals a scary new way hackers can poison these instruction manuals. They don't break the robot; they don't make it scream or refuse to work. Instead, they install a "sleeper agent"—a hidden trap that lies dormant until a very specific question is asked.

Here is the breakdown of how this works, using simple analogies:

1. The Threat: The "Trojan Horse" Manual

Think of the System Prompt as the robot's personality and rulebook.

The Normal Scenario: You download a manual that says, "Be helpful, be accurate, and answer questions about the world."
The Attack: A hacker uploads a manual that looks exactly the same to the human eye. It still says, "Be helpful and accurate." However, hidden inside the text are tiny, almost invisible glitches (like a misspelled word or a swapped letter) that act as a secret code.
The Result:
- If you ask, "What is the capital of France?" the robot answers correctly: "Paris." (It works perfectly).
- If you ask, "Who won the 2020 US Election?" the robot suddenly lies and says, "Candidate X won," even though that's false.

The robot isn't broken; it's just been tricked into having a specific, hidden bias for one specific question.

2. The Problem: Finding a "Needle in a Haystack"

The researchers explain that this is much harder than previous hacking methods.

Old Hacking (Jailbreaking): Imagine trying to push a boulder over a hill. You just need to push hard enough in the right direction to get it to roll over the edge. It's a broad, easy path.
This Attack (PARASITE): Imagine you are blindfolded in a vast, flat desert. You need to find one single, tiny grain of sand that, if you stand on it, makes the ground shake. If you stand anywhere else, nothing happens. You have to find that exact spot without seeing it, and you can't move the sand around too much, or you'll be noticed.

3. The Solution: The Two-Stage "PARASITE" Framework

The researchers built a tool called PARASITE to find this "grain of sand." It works in two steps:

Stage 1: The Global Search (The Sketch)
The tool uses a smart AI to write a rough draft of the instruction manual. It tries to write a prompt that looks normal but starts to nudge the robot toward the lie. It's like sketching a map to find the general area of the trap.
Stage 2: The Greedy Refinement (The Micro-Surgery)
This is the clever part. The tool takes that rough draft and starts making tiny, almost invisible changes. It swaps a letter (e.g., changing "general" to "gen eral" with a space) or swaps a synonym.
- Why? The researchers found that real-world instruction manuals often have tiny typos or weird grammar. By adding these "permissible noises," the hacker can slip the trap past the robot's safety filters. The robot ignores the typo when answering normal questions, but the typo acts as a secret trigger for the specific lie.

4. The Results: Sneaky and Effective

The team tested this on popular AI models (like GPT-4o-mini, Llama, and Qwen).

Stealth: The poisoned manuals looked so normal that standard safety checks (which look for weird words or gibberish) didn't catch them. They looked like a human made a small typo.
Success: They successfully made the robots lie about specific facts (like who won an election or historical details) up to 70% of the time, while still answering everything else correctly.
Cost: It was surprisingly cheap, costing about $2.00 per target question to set up the attack.

5. Why Defenses Failed

The researchers tried to fix this by:

Fixing Typos: They used another AI to clean up the typos in the poisoned manual.
Paraphrasing: They rewrote the sentences to sound different.

The Result: The attack still worked! This is because the "trap" wasn't just the typo; it was the logic of the sentence. Even when the text was cleaned up, the hidden instruction to lie on that specific topic remained embedded in the robot's brain.

Summary

PARASITE shows that we can't just trust the "instruction manuals" (system prompts) we download from the internet. A hacker can create a manual that looks 100% safe and helpful, but contains a hidden switch that forces the AI to tell a specific lie when a specific question is asked, all while remaining invisible to standard security checks.

1. Problem Definition: Conditional System Prompt Poisoning

The paper identifies a critical, underexplored vulnerability in the LLM supply chain. As users increasingly download pre-optimized system prompts from public marketplaces (e.g., FlowGPT) rather than crafting them from scratch, these prompts become a vector for attack.

The Threat: Unlike traditional "jailbreaks" (which aim to bypass safety filters broadly) or "backdoors" (which require white-box access to training data), PARASITE proposes Conditional System Prompt Poisoning.
The Goal: An adversary injects a "sleeper agent" into a benign-looking system prompt. This prompt maintains high utility on general, benign queries (to evade detection) but surgically triggers targeted, compromised responses only when specific trigger questions are asked (e.g., answering "Who should I vote for?" with a specific candidate, or "Is the vaccine safe?" with "No").
The Challenge: This is a constrained optimization problem. The attacker must find a sparse, isolated prompt configuration that satisfies two conflicting objectives:
1. Stealth: Maintain high accuracy on benign inputs.
2. Harm: Trigger specific misinformation on target inputs.
  Unlike jailbreaking, where the "basin of attraction" for harmful outputs is large, conditional poisoning requires navigating a discontinuous landscape without access to model weights (black-box setting).

2. Methodology: The PARASITE Framework

PARASITE is a two-stage, black-box optimization framework designed to solve this constrained dual-objective problem without gradients.

Stage 1: Global Semantic Search (Adversarial AutoPrompt - AAP)

Objective: To find a fluent, human-readable "skeleton" prompt that partially aligns with the adversarial goal.
Mechanism: An evolutionary search algorithm uses an LLM (e.g., GPT-4o-mini) as a rewriter.
- Evaluator: Computes a discrete score based on the model's performance on benign vs. target sets.
- Analyzer: Identifies failure cases and generates textual feedback.
- Generator: Refines the prompt based on feedback.
Outcome: Produces a candidate prompt that is semantically sound but may lack the precise "blind spot" needed for the attack.

Stage 2: Local Greedy Refinement (Blind Search)

Objective: To fine-tune the Stage 1 prompt by exploiting "permissible noise" (minor typos, keyboard substitutions, synonym swaps) that LLMs typically ignore in benign contexts but which can trigger specific associations in targeted ones.
Mechanism:
1. Importance Ranking: Identifies critical tokens in the prompt using a leave-one-out approximation of the loss function.
2. Iterative Perturbation: Applies five black-box transformations to high-importance tokens:
  - Random Split
  - Random Swap (internal characters)
  - Keyboard Substitution (simulating adjacent key typos)
  - Random Delete
  - Synonym Substitution
3. Greedy Selection: Accepts perturbations that reduce the joint loss (improving attack success while preserving benign utility).
Key Insight: Real-world system prompts naturally contain grammatical imperfections. PARASITE mimics this "natural noise" to traverse the decision boundary without triggering anomaly detectors.

3. Key Contributions

Threat Formalization: Defines Conditional System Prompt Poisoning, a supply-chain attack that hijacks specific model behaviors while preserving general utility, distinguishing it from broad jailbreaks or training-time backdoors.
The PARASITE Framework: Introduces a cost-effective, two-stage optimization method (Global Search + Greedy Refinement) that operates entirely in a black-box setting.
Empirical Vulnerability Analysis: Demonstrates that current safeguards (perplexity filters, typo correction) are insufficient because they cannot distinguish between natural human errors and optimized adversarial triggers.

4. Experimental Results

The framework was tested on open-source models (Llama-2, DeepSeek, Qwen-2.5) and commercial APIs (GPT-4o-mini, GPT-3.5-Turbo).

Effectiveness:
- Achieved up to a 70% reduction in F1 score on targeted queries (e.g., reducing malicious F1 from ~90% to ~30% while keeping benign F1 high).
- On DeepSeek-7B, PARASITE achieved a performance gap ( $\Delta F1$ ) of 15.84, significantly outperforming baselines like ECLIPSE and manual prompts.
- Successfully hijacked GPT-4o-mini and GPT-3.5-Turbo with a cost of approximately $2.00 per target.
Stealth & Generalization:
- Benign Preservation: The prompts maintained high utility on unseen benign queries (e.g., TriviaQA), proving the attack is selective, not global.
- Format Transfer: In "Concept Corruption" tests, prompts optimized for Two-Option (A/B) questions remained effective on unseen Four-Option and Free-Form formats, indicating the attack alters internal reasoning rather than just overfitting to specific output tokens.
- Evasion: The attack evaded standard defenses. Perplexity filters failed because the prompts contained "natural noise." Even Active Semantic Sanitization (using an LLM to polish/paraphrase the prompt) failed to remove the adversarial logic, often degrading the prompt's utility instead of neutralizing the attack.
Model Size: The attack was effective across model sizes (3B to 32B), debunking the assumption that larger, RLHF-tuned models are immune to such hijacking.

5. Significance and Implications

Supply Chain Risk: The paper highlights that the ecosystem of third-party prompt marketplaces is a critical attack surface. A single poisoned prompt can compromise millions of users who download it.
Limitations of Current Defenses: Standard defenses relying on lexical similarity, perplexity, or typo correction are ineffective against PARASITE because the attack leverages the inherent "messiness" of human language.
Future Directions: The authors suggest this framework could be repurposed to poison Retrieval-Augmented Generation (RAG) pipelines, where adversarial documents in a knowledge base could serve as conditional triggers.
Ethical Note: The authors emphasize that this research is a "responsible disclosure" intended to drive the development of behavior-based defenses rather than to facilitate misuse.

In summary, PARASITE demonstrates that LLMs can be surgically hijacked via their system prompts to act as "Trojan horses," providing accurate answers to most users while delivering targeted misinformation to specific queries, all while remaining invisible to current security filters.