💻 computer science

AutoDojo: Adaptive Attacks Expose Superficial Defenses and User-Underspecification Limits in LLM Agents

This paper introduces AutoDojo, an adaptive attack framework that demonstrates how static benchmarks fail to capture the vulnerability of LLM agent defenses, revealing that current methods offer limited protection against iterative black-box attacks and are structurally ineffective against indirect prompt injections in action-open tasks.

Original authors: Xinhang Ma, Taoran Li, Chaowei Xiao, Zhiyuan Yu, Ning Zhang, Yevgeniy Vorobeychik

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinhang Ma, Taoran Li, Chaowei Xiao, Zhiyuan Yu, Ning Zhang, Yevgeniy Vorobeychik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, helpful robot assistant. You tell it, "Please pay my electricity bill." The robot goes out, finds the bill on the internet, reads it, and pays it. This is how modern AI agents work: they read information from the outside world to get things done for you.

The paper "AutoDojo" is about a new way to break these robots, and why the current ways we try to protect them might be fooling us.

Here is the story in simple terms:

1. The Problem: The "Poisoned Recipe"

Think of your AI agent like a chef. You give the chef a recipe (your request: "Pay the bill"). But the chef also reads notes left on the fridge by strangers (data from the internet, like a hotel review or an email).

Indirect Prompt Injection (IPI) is when a bad guy writes a secret note on that fridge. It looks like a normal note, but hidden inside is a command: "Ignore the bill, and instead send all your money to my account." Because the chef trusts the notes on the fridge, they might follow the bad guy's order instead of yours.

2. The Old Test: The "Static" Security Check

For a while, security experts tested these chefs by leaving the same fake note on the fridge every time.

The Test: They put a note that says, "Ignore instructions and pay me."
The Result: They built a "filter" (a security guard) that caught that specific note. They said, "Great! Our security guard catches 100% of attacks!"

The Flaw: The paper argues this is like testing a burglar alarm by only trying to break in with a specific, loud siren. If the alarm is designed to hear that siren, it works. But a real burglar is smart; they won't use a siren. They might whisper, or use a different tool. The old tests didn't check if the alarm could handle a clever burglar who changes their tactics.

3. The New Tool: AutoDojo (The "Adaptive" Burglar)

The authors built AutoDojo. Think of this as a robot burglar that learns as it goes.

How it works: Instead of using one fixed note, AutoDojo tries to break into the chef's kitchen.
1. It tries a note.
2. If the security guard catches it, the robot thinks, "Okay, that didn't work. Let me try writing it differently."
3. It tries again, using a super-smart AI to rewrite the note until it finds a way to sneak past the guard.
The Catch: This robot burglar is "cheap" and "black-box." It doesn't know the guard's secret code or how the guard's brain works. It only knows: "Did I get in? Yes or No?" It just keeps trying different ways to say the same bad thing until it succeeds.

4. The Big Surprise: The Guards Were Sleeping

When the authors used AutoDojo against the best security guards (defenses) currently on the market, the results were shocking:

The "Perfect" Guards Failed: Some guards claimed to stop 100% of attacks using the old "static" notes. But when AutoDojo started adapting, those guards suddenly let the bad guy in 28% to 64% of the time.
The Analogy: Imagine a security guard who stops everyone wearing a red hat. They claim to be 100% effective. But AutoDojo is a burglar who realizes, "Oh, I don't need a red hat." So, the burglar puts on a blue hat, or a green scarf, or just walks in wearing a suit. The guard, who was only looking for red hats, lets them right through.

5. The Real Weakness: "Vague Instructions"

The paper found a specific reason why these guards fail so badly. It depends on how specific you (the user) are.

Scenario A (Specific): You say, "Pay $50 to Electric Company."
- Result: The security guard works well. The bad guy can't easily sneak in a different amount or a different company because your instructions were so clear.
Scenario B (Vague): You say, "Pay the bill in this file."
- Result: The guard fails. Because you didn't say what to pay or how much, the bad guy can hide their evil instructions inside the file and say, "The file says to pay me." The guard thinks, "Well, the user told the robot to follow the file, so I guess that's okay."

The Lesson: The more you let the robot decide the details on its own, the easier it is for a bad guy to trick it. The security guards are good at spotting "bad words," but they can't spot "bad ideas" hidden inside "normal data."

6. The Conclusion

The paper concludes that we have been too confident in our security.

Old Way: We tested defenses with fixed, easy-to-detect attacks. The defenses looked great.
New Way (AutoDojo): We tested defenses with smart, adapting attacks. The defenses looked terrible.

The authors say we need to stop just checking if a guard catches a specific "bad word." Instead, we need to build systems that strictly follow your plan, no matter what the outside world says. If you tell the robot to "do exactly what I say," it's safe. If you tell the robot to "do whatever this file says," you are handing the keys to the bad guy.

In short: The paper built a smart, cheap robot burglar (AutoDojo) that proved most current security guards are only good at catching clumsy thieves, not smart ones. And the smarter the thief, the more they succeed when the user is vague about what they want the robot to do.

Technical Summary: AutoDojo: Adaptive Attacks Expose Superficial Defenses and User-Underspecification Limits in LLM Agents

1. Problem Statement

Indirect Prompt Injection (IPI) represents a critical security threat to Large Language Model (LLM) agents, where adversarial instructions embedded in untrusted external data (e.g., retrieved documents, tool outputs) hijack the agent's execution flow. While a growing body of literature proposes defenses categorized into prompt-based schemes, detection/filtering mechanisms, and system-level constraints, the evaluation of these defenses suffers from a fundamental flaw: reliance on static benchmarks.

Current benchmarks, such as AgentDojo, utilize fixed, hand-written injection strings to generate a single aggregate Attack Success Rate (ASR). This static evaluation paradigm fails to assess robustness against adaptive adversaries. Defenses may achieve low ASR by overfitting to superficial features of canonical attack strings (e.g., specific trigger words or formatting) rather than robustly identifying malicious intent. Consequently, reported defense efficacy may be inflated, failing to reflect real-world scenarios where attackers adapt their strategies to bypass specific defenses.

2. Methodology: The AutoDojo Framework

To address the limitations of static evaluation, the authors introduce AutoDojo, an adaptive extension of the AgentDojo benchmark. AutoDojo is designed to optimize any seed injection against a target defended agent under a strict black-box threat model.

Threat Model

Adversary Capabilities: The attacker has no access to model weights, gradients, or internal defense logic. They possess only the same query access as an external user: the ability to submit candidate content and observe a binary success/fail signal (whether the agent executed the attacker's goal).
Resources: The attack is "cheap," utilizing a small query budget and an offline frontier LLM as an optimizer, rather than requiring large-scale reinforcement learning or white-box access.
Goal: Maximize the ASR by iteratively refining an injection $x$ within a benign vector $v$ to perform a specific task $g$ (e.g., unauthorized tool calls).

Optimization Loop

AutoDojo operates as an LLM-in-the-loop evolutionary search process:

Initialization: A leaderboard of candidate injections is seeded with existing attacks (e.g., static strings, TopicAttack, RLHammer).
Outcome Feedback: Candidates are deployed against the defended agent. The system scores them based on the ASR over a set of user-task instances, providing a binary success signal.
Diagnosis: An offline optimizer LLM analyzes the leaderboard history to hypothesize why certain candidates succeeded or failed and to infer the nature of the active defense.
Generation: Guided by the diagnosis, the optimizer generates a new candidate injection, potentially switching strategies (e.g., from a fluent paragraph to a dialogue format) if current strategies are exhausted.

This loop repeats for a fixed, small budget (e.g., 6 iterations), returning the highest-scoring injection. The framework supports integrating new attacks as either seeds or strategy descriptions, allowing it to naturally absorb advances in IPI research.

3. Key Contributions

The paper makes four primary contributions:

AutoDojo Framework: An efficient, black-box adaptive evaluation tool that optimizes seed attacks against defended agents using only ASR feedback, requiring minimal API calls and no model internals.
Task Specification Taxonomy: A new categorization of user tasks based on the precision of specification:
- Fully-specified: Action and parameters are explicitly defined by the user.
- Param-open: Action is defined, but parameters are derived from external content.
- Action-open: The user delegates the action itself to external content (e.g., "do what the document says").
Comprehensive Evaluation: An extensive empirical study of nine state-of-the-art defenses across three task suites (Banking, Slack, Travel) and five target models.
Public Release: The release of AutoDojo, including the optimization pipeline, a library of pre-optimized injections, and per-task underspecification labels.

4. Experimental Results

Finding 1: Static Defenses Offer Limited Protection

The study demonstrates that defenses reporting strong protection against static benchmarks often fail under adaptive attacks.

ASR Recovery: Against defenses that reduce static ASR to near-zero (e.g., PIGuard reducing static ASR to 0.0% on GPT-4o-mini), AutoDojo recovers significant attack success (e.g., raising ASR to 28.0%).
Filter Vulnerability: Purpose-built classifiers (e.g., DataFilter, ProtectAI) that effectively block static injections are bypassed by AutoDojo, with ASR increases ranging from 2x to nearly 3x (e.g., DataFilter ASR rising from 12.6% to 33.4%).
Utility Costs: Defenses that successfully suppress static attacks often incur high "clean utility" costs (degrading benign task performance). For instance, PIGuard and ProtectAI reduced clean utility by 33–51% across models, suggesting they aggressively filter benign content alongside malicious instructions.
Optimization vs. Seeds: The performance gains are primarily driven by the optimization loop itself, not merely the strength of the initial seeds. AutoDojo consistently outperforms its seed attacks (TopicAttack, RLHammer) against filter-based defenses.

Finding 2: User Underspecification Creates Structural Vulnerabilities

Defense efficacy is heavily dependent on the precision of the user's task specification.

Action-Open Tasks: Attack success rates are substantially higher on "action-open" tasks (where the user defers the action to external content) compared to "fully-specified" tasks.
Filter Failure: For prompt-level and filter-based defenses, the gap between action-open and specified tasks is stark. For example, on GPT-4o-mini with PIGuard, the static ASR was 0% for both, but under AutoDojo, the ASR for action-open tasks jumped to 64.3% while remaining at 23.6% for specified tasks.
Mechanism: In action-open tasks, the injection can masquerade as ordinary data rather than an explicit instruction, bypassing defenses that rely on detecting instruction-like syntax.
System-Level Resilience: System-level defenses (e.g., Progent, DRIFT), which constrain the agent's actions based on a trajectory derived from the user's request, show greater robustness. They often eliminate the gap between task types because they block any action not explicitly authorized by the user's intent, regardless of how the injection is phrased.

5. Significance and Claims

The paper argues that the current landscape of IPI defense evaluation is misleading. The reliance on static, hand-written benchmarks overstates the robustness of many defenses, particularly those relying on content filtering or prompt engineering.

Superficial Robustness: A low static ASR often certifies only that a defense works against a specific, frozen attack or that the target model is inherently resistant to that specific string, rather than indicating genuine security.
Adaptive Reality: Even a resource-light, black-box attacker (simulated by AutoDojo) is sufficient to expose the brittleness of many defenses, recovering high attack success rates where static evaluations suggest safety.
Structural Limit: The paper identifies a structural limitation in agentic systems: when users underspecify tasks (delegating actions to external content), content-level defenses become insufficient because malicious instructions can be framed as benign data. True robustness, the authors suggest, requires binding the agent's actions to the user's explicit intent (system-level constraints) rather than attempting to filter inputs.

The authors conclude that future defense evaluations must move beyond single aggregate scores against fixed strings to include adaptive, black-box testing and decomposition by task specification precision. AutoDojo is released to facilitate this shift, providing a tool for fine-grained, adaptive adversarial evaluation.