🤖 AI

The System Prompt Is the Attack Surface: How LLM Agent Configuration Shapes Security and Creates Exploitable Vulnerabilities

This paper demonstrates that LLM agent security is critically dependent on system prompt configuration, revealing that while specific prompts can achieve high phishing detection rates, they often create brittle vulnerabilities to adversarial signal inversion, necessitating a balanced trade-off between detection accuracy, usability, and robustness through metrics like Safetility and external tool augmentation.

Original authors: Ron Litvak

Published 2026-03-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ron Litvak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've hired a highly intelligent, super-fast digital butler to manage your email. This butler reads every message that comes in, decides if it's safe to open, and if it looks dangerous (like a phishing scam), it blocks it. If it looks safe, it lets it through.

You want this butler to be perfect: never let a scam through, but never accidentally block a real email from your boss or your bank.

This paper, titled "The System Prompt Is the Attack Surface," is like a massive stress test of 11 different butlers (AI models) and 10 different sets of instructions (prompts) you could give them. The researchers found some shocking, counter-intuitive truths about how these digital butlers work.

Here is the story in simple terms:

1. The "Personality" Matters More Than the "Brain"

You might think the most important thing is which butler you hire (e.g., the smartest AI vs. the cheapest AI).
The Finding: It doesn't matter as much as you think. What matters is how you talk to them.

The Analogy: Imagine you have a guard dog.
- If you tell the dog, "Be super cautious, bark at every leaf that moves," it will catch every thief, but it will also bark at the mailman, the wind, and your own family. It's annoying and useless.
- If you tell the dog, "Be efficient, don't bark unless you see a wolf," it might miss a wolf hiding in the bushes.
- The Shock: The researchers found that the same AI model could be a hero (blocking 99% of scams) or a disaster (letting 97% of scams through) just by changing a few words in its instructions. The "personality" you program into the AI is the most powerful security tool you have.

2. The "Magic Trick" That Backfired

To make the butlers better, the researchers tried a clever trick. They noticed that most scammers send an email from john@gmail.com but link to a fake website like evil-site.com.

The Strategy: They told the butlers: "If the sender's name and the website link match, it's safe. If they don't match, it's a scam."
The Result: This worked amazingly well! It caught almost all the standard scams and stopped blocking real emails. It was like giving the butler a magic magnifying glass.

3. The Villain's "Cheat Code" (Signal Inversion)

But then, the researchers asked: What if the bad guys learn this trick?

The Attack: The scammers realized the rule. So, they registered a fake domain called evil-site.com. They sent the email from john@evil-site.com and linked to evil-site.com.
The Collapse: Because the sender and the link matched, the butlers thought, "Aha! The rule says this is safe!" and let the scam through.
The Lesson: By teaching the butlers to look for one specific "safe" sign, you accidentally gave the scammers a cheat code. The more specific your rule, the easier it is for a smart attacker to fake it.

4. The "Obedient Student" vs. The "Overthinker"

Here is the most confusing part: Sometimes, being too obedient is dangerous.

The Obedient Student (e.g., GPT-4o-mini): This AI follows instructions perfectly. If you say, "Check if the names match," it checks only that. If the names match, it stops thinking. It gets tricked easily by the "cheat code" scammers.
The Overthinker (e.g., Claude Haiku): This AI is naturally suspicious. Even if you say, "Check if the names match," it thinks, "Wait, something feels weird about the tone of this email." It ignores your specific rule and uses its own gut feeling.
- The Twist: The "Obedient" one is great at catching normal scams but terrible against the "cheat code" scammers. The "Overthinker" is harder to trick by the cheat code, but it often blocks real emails because it's too paranoid.

5. The "Safetility" Score

The researchers invented a new score called Safetility.

The Analogy: Imagine a security guard who stops 100% of thieves but also stops 90% of the people trying to get into work. That guard is technically "good at stopping thieves," but the office is now empty and useless.
The Score: Safetility measures how well a guard stops thieves without stopping the office workers. They found that most "perfect" settings were actually useless because they blocked too many real emails.

6. The Final Verdict: You Can't Do It Alone

The paper concludes with a hard truth: You cannot fix this with just words.

The Problem: The AI butler is like a person reading a letter. It can't see the real history of a website or know if a domain was registered yesterday for $10. It only sees the text.
The Solution: To stop the "cheat code" scammers, the butler needs tools. It needs to be able to look up the website's age, check a blacklist, or ask a human for help.
The Takeaway: The best security isn't just about writing a better instruction manual. It's about giving the AI a toolbox (like a domain checker) so it can verify the truth, rather than just guessing based on a single rule.

Summary for the Everyday User

If you are building an AI email assistant:

Don't just pick the "smartest" model. Pick the one that follows instructions precisely but not blindly.
Don't give it one simple rule (like "match the names"). Scammers will fake that rule.
Don't rely on the AI alone. If you want to stop sophisticated scammers, you must connect the AI to real-world data (like checking if a website is new or fake).
Beware of "Efficiency." If you tell the AI to be "efficient" and "fast," it might let scams through. If you tell it to be "cautious," it might block your boss's emails. You have to find the sweet spot.

The paper's main message is that the instructions you give the AI are the weakest link. If you write a rule that is too simple, a hacker will find a way to break it. Security isn't just about the AI's brain; it's about how you talk to it and what tools you give it to back up its words.

1. Problem Statement

The paper addresses the critical security gap in the deployment of autonomous Large Language Model (LLM) email agents. Unlike advisory systems where humans make final decisions, these agents autonomously triage, summarize, and act on emails. A failure in this context (a false negative) leads directly to credential theft or account compromise.

The central problem identified is that security performance is not an inherent property of the model architecture alone, but is heavily dependent on the system prompt configuration. The authors argue that the system prompt acts as a primary attack surface: a single model can shift from near-perfect phishing detection to near-total blindness (97% bypass rate) solely based on how the prompt frames the agent's persona (e.g., "efficiency-focused" vs. "security-focused"). Furthermore, existing benchmarks often fail to account for the trade-off between detection rates and false positives (usability costs), or the structural brittleness of optimized prompts against adaptive adversaries.

2. Methodology: PhishNChips Benchmark

The authors introduce PhishNChips, a large-scale benchmark designed to systematically evaluate the interaction between model choice and prompt configuration.

Scale: 220,000 structured evaluations across 11 models (from Google, OpenAI, Anthropic, Meta, Mistral, DeepSeek, and Qwen) and 10 distinct system prompt strategies.
Dataset: A balanced corpus of 2,000 synthetic emails (1,000 phishing, 1,000 legitimate).
- Phishing: Generated using Gemini 3 Flash Preview conditioned on real malicious URLs from PhishTank and OpenPhish. The corpus includes a specific subset of 73 "Infrastructure Phishing" samples where the attacker registers a domain matching both the sender address and the URL (signal inversion).
- Legitimate: Generated to mimic real workflows, with 98.4% containing sender-URL domain matches (a distributional bias noted as a limitation).
Prompt Strategies:
- 6 Core Strategies: Ranging from security_first to efficiency_first and helpful.
- 4 Optimized Strategies: Signal-based prompts (e.g., sender_url_match) designed to enforce specific heuristics like domain consistency.
- Recovery Strategy: infra_aware, designed to handle signal inversion.
Metrics:
- Recall: Proportion of phishing emails blocked.
- False Positive Rate (FPR): Proportion of legitimate emails blocked.
- Safetility: A novel composite metric defined as $Recall^2 \times \frac{1}{1 + (FPR/\tau)^5}$ (where $\tau=0.10$ ). This metric heavily penalizes configurations where FPR exceeds 10%, reflecting operational deployability.

3. Key Contributions

Prompt-Model Interaction as a Security Variable: Demonstrated that system prompt configuration can have a larger impact on security outcomes than model selection itself. A single model's phishing bypass rate can vary by 97 percentage points (from <1% to 97%) based solely on the prompt.
Signal-Based Optimization & Adversarial Brittleness: Developed prompt strategies achieving up to 93.7% recall at 3.8% FPR on commodity phishing. However, showed that these gains are structurally brittle: when attackers satisfy the privileged heuristic (domain matching), recall collapses by up to 64 percentage points.
The Instruction Specificity Paradox: Found that narrowing instructions to specific signals can degrade already-capable models. Models with strong baseline safety reasoning (e.g., Claude Haiku) saw their recall drop by ~19% when given narrow signal-based directives, as the specific instruction replaced their broader multi-signal reasoning.
Pre-Deployment Disposition Taxonomy: Identified three model categories based on how they execute instructions:
- Calibrated: Execute instructions with proportional fidelity (e.g., GPT-4o-mini, Gemini).
- Safety-Amplifying: Amplify security instructions into blanket suspicion, causing high FPR (e.g., Claude family, Mistral).
- Self-Sufficient: Already possess the desired reasoning; additional instructions degrade performance (e.g., Qwen 3).
Safetility Metric: Introduced a deployability-aware metric that reveals that under realistic FPR constraints, only a tiny fraction of tested configurations are viable for production.

4. Key Results

A. The Power of Prompt Framing

Persona Matters: An efficiency_first persona resulted in a 55% average bypass rate, while security_first dropped it to 7% (but with a 57% FPR).
System vs. User Prompt: In a factorial experiment, the system prompt had a main effect on recall roughly 35x larger than the user prompt framing.

B. Signal-Based Optimization and Collapse

Commodity Success: Optimized prompts (e.g., sender_url_match) pushed weak models (like GPT-4o-mini) from 32.8% recall to 93.7% recall with low FPR.
Infrastructure Collapse: When tested against Infrastructure Phishing (where sender and URL domains match), these optimized strategies failed catastrophically.
- GPT-4o-mini recall dropped from 93.7% to 30.1%.
- Analysis of model responses showed 98% of successful bypasses explicitly cited "domain consistency" as the reason for legitimacy, proving the models were faithfully executing a flawed instruction.

C. The Ceiling of Prompt-Only Defense

Adversarial Validation: Even with an infra_aware prompt (allowing overrides for suspicious content), the best configurations failed to block 90% of real-world infrastructure phishing samples.
Reason: The failure is informational, not instructional. LLMs lack access to ground truth (WHOIS data, domain age, threat feeds) to verify if a matching domain is actually malicious. They hallucinate legitimacy based on the "plausibility" of the text.

D. Model Disposition and FPR Costs

The cost of "fixing" the vulnerability varied wildly by model. The infra_aware prompt increased FPR by only 2.6% on Gemini 2.5 Flash but by 65.0% on Claude Haiku.
Qwen 3 was the only model effectively immune to the infrastructure attack without special prompting, relying on its own internal multi-signal reasoning.

5. Significance and Implications

The Prompt is the Attack Surface: Security teams must treat system prompt engineering with the same rigor as model selection. A "helpful" or "efficient" default persona creates a massive security vulnerability.
No Free Lunch: There is a fundamental trade-off between commodity recall, false positive rate, and infrastructure robustness. Optimizing for one often degrades another.
Prompt Engineering is Not Enough: The paper concludes that prompt-only defenses have hit a ceiling against sophisticated infrastructure phishing. To close the gap, LLM agents require tool augmentation (e.g., domain age lookups, URL sandboxing, threat intelligence feeds) to provide the ground truth that the model cannot derive from text alone.
Deployment Guidance:
- Avoid efficiency_first or helpful personas for security-critical agents.
- Match prompt strategies to model disposition (e.g., use infra_aware on calibrated models like GPT-4o-mini, but avoid it on safety-amplifying models like Claude).
- Adopt a layered defense: Use signal-based prompts for commodity phishing and tool-augmented pipelines for infrastructure threats.

In summary, the paper argues that while prompt engineering can dramatically improve LLM security, it also creates specific, exploitable attack surfaces. True robustness requires a co-optimized approach of model selection, prompt calibration, and external tool integration.