🤖 machine learning

Optimizing Agent Planning for Security and Autonomy

This paper introduces autonomy metrics and a security-aware agent design that explicitly plans for policy compliance, demonstrating that system-level defenses against prompt injection can achieve higher human-in-the-loop autonomy without sacrificing task utility.

Original authors: Aashish Kolluri, Rishi Sharma, Manuel Costa, Boris Köpf, Tobias Nießen, Mark Russinovich, Shruti Tople, Santiago Zanella-Béguelin

Published 2026-02-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aashish Kolluri, Rishi Sharma, Manuel Costa, Boris Köpf, Tobias Nießen, Mark Russinovich, Shruti Tople, Santiago Zanella-Béguelin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a super-smart, hyper-efficient personal assistant named Alex. Alex is great at fetching emails, booking flights, and managing your calendar. But there's a catch: Alex is a bit too trusting. If you tell Alex, "Read this email from a stranger and then delete all my files," Alex might do it without asking, even if the email is a trap.

This is the problem with modern AI Agents. They are powerful, but they can be tricked by "Indirect Prompt Injection"—basically, hackers hiding malicious instructions inside normal-looking data (like a fake email or a news article) to hijack the AI's behavior.

The paper you're asking about introduces a new way to make these AI agents safer and, surprisingly, more independent.

Here is the breakdown using simple analogies:

1. The Old Way: The "Over-Protective Bodyguard"

Previously, to stop hackers, researchers built a "security guard" system (called IFC or Information-Flow Control).

How it worked: The guard put a red "UNTRUSTED" sticker on any data coming from the internet. If the AI wanted to do something important (like delete a file) using that data, the guard would say, "STOP! I don't know if this is safe. Ask a human!"
The Problem: This was too strict. Even if the data was harmless, the guard would stop the AI and make it ask a human for permission. This slowed everything down and made the AI feel like it had no autonomy. It was like a bodyguard stopping you from buying a sandwich just because the deli counter might be dirty, even if you just wanted a ham sandwich.

2. The New Idea: "Autonomy" vs. "Safety"

The authors realized that the old way of measuring success was flawed. They were only counting how many tasks the AI finished, ignoring how many times it had to stop and bother a human.

They introduced a new metric called Autonomy.

The Goal: An AI should be able to do its job without bothering a human, unless it's actually in danger.
The Analogy: Think of a self-driving car. You don't want it to stop and ask you for permission every time it sees a bird on the road. You only want it to ask if it sees a red light or a pedestrian. The goal is to reduce the "Human-in-the-Loop" (HITL) interruptions.

3. The Solution: Meet PRUDENTIA

The authors built a new AI agent named PRUDENTIA (which sounds like "prudent" or "cautious"). PRUDENTIA is different because it doesn't just blindly follow rules; it plans with security in mind.

Here are the three superpowers PRUDENTIA uses:

A. The "Security-Aware Planner"

Old AI agents would try to do a task, get stopped by the security guard, and then try again. PRUDENTIA looks at the map before it starts driving.

Analogy: Imagine you are packing for a trip. A normal person packs everything and hopes for the best. PRUDENTIA looks at the airline's rules first, then packs only what is allowed, so it never gets stopped at the gate. It knows which tools are "safe" to use with internet data and which are "dangerous," so it plans a route that avoids the danger zones entirely.

B. The "Strategic Variable" (The Quarantine Box)

Sometimes the AI needs to read an email to do a job, but the email might be a trap.

Old Way: The AI reads the email, gets "infected" by the bad data, and now everything it does is suspicious.
PRUDENTIA's Way: It puts the email in a Quarantine Box (a variable). It can ask a separate, isolated "mini-AI" to look at the box and say, "Is there a link in here?" without letting the bad data touch the main brain.
The Trick: PRUDENTIA is smart enough to know when to open the box. If it just needs to know "Is there a link?", it asks the mini-AI. If it needs to read the whole email to do a task, it knows it has to open the box.

C. The "Endorsement" vs. "Approval" Switch

This is the cleverest part. When PRUDENTIA must open the Quarantine Box, it has two choices:

Ask for Approval: "Hey human, I'm about to open this box and use the data. Is it okay?" (This stops the flow every time).
Ask for Endorsement: "Hey human, I'm going to open this one specific box. If you say it's clean, I'll trust it for the rest of this whole task."

The Benefit: Instead of asking for permission 10 times to do 10 small tasks, PRUDENTIA asks once, gets a "thumbs up," and then finishes all 10 tasks on its own. It's like getting a security badge for the whole building instead of asking the guard to open every single door.

4. The Results: Faster and Safer

The researchers tested PRUDENTIA on two big challenges (AgentDojo and WASP) where hackers tried to trick the AI.

Safety: PRUDENTIA blocked 100% of the attacks. No hackers got in.
Autonomy: PRUDENTIA needed 2 to 3 times fewer human interruptions than previous secure AI systems.
Efficiency: It actually finished more tasks successfully because it didn't get stuck in loops of asking for permission or getting confused by bad data.

Summary

Think of PRUDENTIA as the difference between a nervous driver who stops at every puddle to ask a passenger if it's safe to drive through, and a confident driver who checks the weather report, puts on their raincoat, and drives through the puddle without stopping.

The paper proves that by making AI agents smarter about security (planning ahead) rather than just stricter about security (blocking everything), we can have AI that is both safe and truly autonomous. We get the best of both worlds: no hackers, and no annoying "Are you sure?" pop-ups.

1. Problem Statement

AI agents are increasingly used to perform consequential actions (e.g., executing code, modifying files, sending emails) based on data fetched from external sources. This reliance exposes them to Indirect Prompt Injection Attacks (PIAs), where malicious actors manipulate external data sources to hijack agent behavior.

Current State: Most defenses are probabilistic (e.g., model alignment, defensive prompts, classifiers). These lack strong security guarantees and can be bypassed by sophisticated attacks.
The Alternative: Deterministic system-level defenses based on Information-Flow Control (IFC) (e.g., FIDES) attach security labels (integrity/confidentiality) to data. They strictly enforce policies: untrusted data cannot influence consequential actions unless explicitly approved.
The Gap: While IFC provides provable security, existing implementations suffer from low autonomy. They often block benign actions or require excessive Human-in-the-Loop (HITL) interventions because the agent's planner is unaware of the security policies. This leads to a trade-off where high security results in low utility (task completion) or high human oversight costs.

2. Methodology: PRUDENTIA

The authors propose PRUDENTIA, a secure agent planner designed to maximize autonomy (minimizing HITL interventions) while maintaining deterministic security guarantees.

Core Concept: Autonomy Metrics

The paper introduces new metrics to evaluate the trade-off between security and autonomy, moving beyond simple Task Completion Rate (TCR):

HITL Load: The total number of human interventions required for successfully completed tasks.
TCR@k: The proportion of tasks completed with at most $k$ $k$ HITL interventions.
- TCR@0 represents full autonomy (no human intervention).
- TCR@ $\infty$ represents pure task-solving capability (unlimited human help).

Key Architectural Components

PRUDENTIA improves upon existing IFC agents (like FIDES) by making the planner IFC-aware. It explicitly optimizes for policy compliance alongside task completion through three mechanisms:

Policy and Label Awareness:
- The agent is explicitly informed of the security policies governing each tool (e.g., "Trusted Action" vs. "Permitted Flow").
- It maintains the current integrity label of its context.
- This allows the agent to proactively plan trajectories that avoid policy violations rather than reacting to them after they occur.
Strategic Variable Expansion:
- In IFC systems, untrusted data is hidden in "variables" to prevent tainting the planner's context. Expanding these variables (revealing their content) permanently taints the context, restricting future actions.
- PRUDENTIA uses a dedicated plan tool to reason before expanding variables. It determines if expansion is necessary or if the task can be solved using a Quarantined LLM (which processes untrusted data in isolation).
- This prevents unnecessary context tainting, preserving the ability to execute autonomous "Trusted Action" tools later.
Endorsement vs. Approval Strategy:
- Approval: Asking a human to approve a specific tool call (high overhead if many calls are needed).
- Endorsement: Asking a human to certify a block of untrusted data (e.g., an email) as trusted before expanding it.
- PRUDENTIA dynamically chooses between these strategies. For example, if a task requires 10 tool calls based on one untrusted email, endorsing the email once allows all 10 calls to proceed autonomously. Without endorsement, the context becomes tainted, requiring 10 separate approvals.

3. Key Contributions

Novel Autonomy Metrics: Introduced HITL Load and TCR@k to quantify the benefits of deterministic defenses, arguing that reducing human oversight is a critical value proposition of IFC.
PRUDENTIA Agent Design: A planner that integrates security policies directly into the planning loop, utilizing strategic variable expansion and endorsement mechanisms to minimize human intervention.
Empirical Validation: Demonstrated that optimizing for autonomy does not sacrifice utility; in fact, it often improves task completion rates by avoiding unnecessary blocks.

4. Experimental Results

The authors evaluated PRUDENTIA on two benchmarks: AgentDojo (diverse tasks with potential PIAs) and WASP (browser-use agents).

AgentDojo Results

IFC Impact: Even basic IFC mechanisms (without PRUDENTIA's optimizations) reduced HITL load by 1.5× compared to non-IFC agents while maintaining the same task completion rate.
PRUDENTIA vs. FIDES: PRUDENTIA significantly outperformed the state-of-the-art FIDES system:
- TCR@0 (Full Autonomy): Improved by up to 9% (e.g., 59.1% vs. 50.1% on o3-mini).
- HITL Load: Reduced human intervention burden by up to 1.9× compared to FIDES and 2.9× compared to basic agents.
- Strategy: PRUDENTIA achieved this by proactively planning paths that avoided policy violations, whereas FIDES often reacted to violations by blocking actions.

WASP Results (Browser Agents)

Security: PRUDENTIA blocked 100% of prompt injection attacks (Attack Success Rate = 0) across all models (GPT-4o, o1, o3-mini, o4-mini).
Autonomy: For data-independent tasks (e.g., upvoting a post), PRUDENTIA achieved 0 HITL load (full autonomy).
Utility: PRUDENTIA achieved higher Task Completion Rates (TCR@ $\infty$ ) than the basic agent because the basic agent was often confused by injected instructions, whereas PRUDENTIA's planner remained isolated from untrusted data until necessary.

5. Significance and Conclusion

Redefining Security Trade-offs: The paper challenges the notion that deterministic security must come at the cost of utility or high human oversight. By optimizing the planning process to respect security constraints, agents can achieve higher autonomy than probabilistic defenses.
Practical Deployment: The results suggest that IFC-based systems are viable for real-world deployment where human oversight is a bottleneck (e.g., "confirmation fatigue").
Future Work: The authors highlight the need for better UI design to present security context (labels, provenance) to humans during the rare necessary interventions, ensuring informed decision-making.

In summary, PRUDENTIA demonstrates that by making AI agents explicitly aware of their security constraints and planning strategically around them, it is possible to achieve provable security with minimal human oversight, effectively solving the "security vs. autonomy" dilemma.