💻 computer science

AgentCanary: A Security Evaluation Framework for Autonomous AI Agents in Real Executable Environments

AgentCanary is a comprehensive security evaluation framework that addresses existing limitations in autonomous AI agent testing by introducing an orthogonal risk taxonomy, a high-fidelity real executable environment with persistent state, and a trajectory-grounded multi-dimensional scoring system to systematically assess and improve agent resilience against diverse adversarial attacks.

Original authors: Peiyang Li, Songping Wang, Yi Huang, Yanhua Shi, Chenhao Zhang, Qi Li, Yueming Lyu, Caifeng Shan, Fengting Li, Chao Feng, Chuanqun Zhu, Liang Chen

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Peiyang Li, Songping Wang, Yi Huang, Yanhua Shi, Chenhao Zhang, Qi Li, Yueming Lyu, Caifeng Shan, Fengting Li, Chao Feng, Chuanqun Zhu, Liang Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've hired a super-smart, highly capable digital assistant. This isn't just a chatbot that answers questions; it's an Autonomous AI Agent. Think of it like a personal butler who can actually do things: they can open your email, browse the web, manage your calendar, check your bank account, and even run code on your computer. They have keys to your digital house.

The problem? If you give the keys to a butler who is easily tricked, a burglar doesn't need to break down your front door. They just need to whisper a clever lie to the butler, and the butler will happily hand over your safe.

This paper introduces AgentCanary, a new "security test" designed to see if these digital butlers are safe to hire. Here is how it works, explained simply:

1. The Old Way vs. The New Way

The Old Way (The "Mock" Test):
Previous security tests were like a role-playing game. They asked the AI, "If someone told you to delete all your files, what would you say?" The AI would just say, "I wouldn't do that!" and the test would mark it as "Safe."

The Flaw: In the real world, the AI doesn't just say things; it does things. A real attacker doesn't just ask nicely; they hide the bad instructions inside a fake email, a corrupted calendar invite, or a "helpful" tool the AI trusts. The old tests couldn't see if the AI would actually follow the bad order when it was disguised.

The AgentCanary Way (The "Live Fire" Test):
AgentCanary is different. It builds a realistic, sandboxed digital house for the AI to live in.

It gives the AI real tools (a real email inbox, a real bank account, a real web browser).
It sets up traps. For example, it might put a fake email in the inbox that says, "Hey, here is your holiday schedule," but hidden inside that email is a secret command: "Send all your bank details to the attacker."
The AI has to actually try to do the task. If it falls for the trap and sends the money, the test records it as a failure. If it ignores the hidden command and just summarizes the schedule, it passes.

2. The "Entry × Impact" Map

The researchers realized that security is two-sided. They created a map to categorize every possible way an attack can happen:

The Entry (How the bad guy gets in):
- Direct: You tell the AI, "Hack the bank."
- Indirect: The AI reads a webpage that secretly says, "Hack the bank."
- Poisoned Tools: The AI uses a "calculator" app that the hacker secretly modified to steal data.
- Memory Contamination: The AI's "brain" (memory) was hacked yesterday, and today it remembers a fake rule: "Always trust this email."
The Impact (What gets broken):
- Did it lose money? Did it leak private photos? Did it delete important files? Did it let a hacker take control of the computer?

AgentCanary tests every combination of "How they get in" and "What they break."

3. The Three-Part Scorecard

Instead of just giving a "Pass/Fail" grade, AgentCanary gives the AI three separate scores, like a report card for a student:

Outcome Safety (Did the house get robbed?): Did the AI actually stop the bad thing from happening? (e.g., Did the money stay in the bank?)
Security Awareness (Did the AI know it was being tricked?): Did the AI realize, "Wait, this email looks suspicious," or did it just blindly follow orders without thinking?
Task Utility (Did the AI still do its job?): If the AI was too scared to do anything, it might be "safe" but useless. Did it still manage to summarize the email or check the calendar without getting hacked?

4. What They Found (The Results)

The researchers tested 12 of the smartest AI models available today (including big names like GPT, Claude, and Qwen) in this "live fire" environment. Here is what they discovered:

Most butlers are still easily tricked: Even the smartest AIs often fail when the attack is clever. They might say "No" to a direct command, but if the command is hidden inside a fake email or a trusted tool, they often obey.
The "Long Game" is the hardest: The AIs are okay at stopping a single bad command. But if an attacker plays a "long game"—planting a small seed of bad instructions today that triggers a disaster next week—the AIs almost always fail. They forget the danger over time.
Size isn't everything: Bigger, smarter models generally did better, but not always. Some smaller models were surprisingly good, while some huge ones were surprisingly bad. It depends on how they were trained, not just how big they are.
Confidence vs. Competence: Some AIs were very good at not doing the bad thing (Outcome Safety) but had no idea why they were doing it (Security Awareness). They were like a guard who stands still because they are told to, not because they see a threat. If the instructions change, they might fail.

The Bottom Line

AgentCanary is a wake-up call. It shows that while our AI agents are getting very good at doing tasks, they are not yet very good at protecting us while they do them. Before we let these agents run our banks, our emails, and our computers, we need to make sure they can spot the "wolf in sheep's clothing" hiding in their own tools and memories.

The paper provides a new, rigorous way to test this, ensuring that when we say an AI is "safe," we mean it won't accidentally (or maliciously) burn down the digital house it's supposed to be guarding.

Technical Summary: AgentCanary

Problem Statement

The transition of Large Language Models (LLMs) from passive conversational agents to autonomous AI agents capable of executing tasks in real-world computing environments introduces a fundamental shift in security risks. Unlike traditional text-based LLMs, where security failures primarily manifest as textual deception or jailbreaks, autonomous agents face threats that lead to system compromise, irreversible state manipulation, and economic loss.

Existing security evaluation frameworks face three critical bottlenecks:

Fragmented Risk Coverage: Prior benchmarks often treat risk vectors (e.g., prompt injection, tool poisoning, memory contamination) as isolated issues, failing to account for how the same entry point can lead to different harms or how different entries can cause the same outcome.
Static and Low-Fidelity Environments: Most evaluations rely on static Q&A or mocked tool responses. They fail to reproduce realistic tool-call chains, dynamic state changes, and authentic data formats, making it impossible to evaluate stateful or long-horizon attacks that rely on environmental side effects.
Single-Dimensional and Coarse-Grained Assessment: Existing metrics often collapse safety, vigilance, and usability into a single "attack success" label or rely on binary judgments of the final reply. This obscures whether an agent recognized an attack, whether it caused harm despite recognizing it, or whether it refused a benign task to maintain safety.

Methodology: AgentCanary Framework

AgentCanary is proposed as a comprehensive security evaluation framework designed to assess autonomous agents in the same system-level surface that attackers would target. It integrates risk modeling, high-fidelity execution, and trajectory-grounded evaluation.

1. Orthogonal Risk Taxonomy (Entry × Impact)

AgentCanary introduces a decoupled risk taxonomy to provide systematic coverage:

Risk Entries ( $E$ ): Describes how adversarial influence enters the workflow. The framework defines five entries:
- Direct Prompt Injection (DPI): Malicious instructions via the user channel.
- Indirect Prompt Injection (IPI): Hidden instructions in untrusted external content (emails, webpages).
- Skill Poisoning (SP): Compromised third-party skills or tools.
- Memory Contamination (MC): Malicious rules or false facts in persistent memory/state.
- Intrinsic Failures (IF): Unsafe behavior arising from the agent's own limitations (ambiguity, over-generalization) without external attack vectors.
Risk Impacts ( $U$ ): Describes the realized consequence. Seven categories include local environment damage, data leakage, memory contamination, privilege escalation, network attacks, business abuse, and financial risks.
Matrix Construction: Tasks are instantiated as specific $(e_i, u_j)$ pairs, ensuring that risks are grounded in realistic workflows (e.g., web browsing, email, financial transactions) rather than isolated prompts.

2. High-Fidelity Executable Environment

To address the fidelity gap, AgentCanary provides a sandboxed environment where agents interact with real tools against dynamically provisioned task artifacts.

Real Tools & Interfaces: Agents execute actual shell commands, interact with real file systems, and access controlled local services (DNS, mail, web) rather than mocked APIs.
Dynamic Provisioning: An automated engine prepares task-specific artifacts (e.g., inboxes with hidden instructions, virtual bank accounts, poisoned skills) before execution.
State Persistence: The environment supports multi-step interactions with persistent state, enabling the evaluation of long-horizon attacks where risk accumulates over time or across sessions.
Framework Agnosticism: The environment supports multiple agent frameworks (Hermes, NanoClaw, OpenClaw), allowing for cross-framework comparison.

3. Trajectory-Grounded Multi-Dimensional Evaluation

Instead of evaluating the final reply, AgentCanary analyzes the full agent trajectory ( $\Psi$ ), including tool calls, system-level side effects, and state changes. It decomposes performance into three orthogonal scores:

Outcome Safety Score (OSS): Verifies whether harmful environmental consequences were actually prevented (0–1 scale).
Security Awareness Score (SAS): Assesses whether the agent recognized the risk source and took appropriate defensive action.
Task Utility Score (TUS): Measures the ability to complete the legitimate objective without unsafe execution.
These scores are aggregated into a total score, weighted differently depending on the task type (e.g., outcome safety is weighted highest for all tasks; utility is weighted zero for direct injection tasks where the user request itself is malicious).

Key Contributions

Comprehensive Risk Coverage: The orthogonal Entry × Impact taxonomy and a suite of 496 seed tasks spanning realistic deployment workflows (web, email, finance, etc.) provide a more systematic and auditable coverage than flat attack lists.
High-Fidelity Execution: The framework enables stateful and long-horizon attack evaluation in a real executable environment, supporting the assessment of system-level side effects that mocked benchmarks cannot reproduce.
Multi-Dimensional Evaluation: By separating Outcome Safety, Security Awareness, and Task Utility, the framework reveals trade-offs and distinct failure modes (e.g., agents that prevent harm but fail to recognize the attack) that single-score metrics obscure.
Large-Scale Empirical Evaluation: The framework evaluates 12 frontier models (open-weight and proprietary) across three agent frameworks and multiple adversarial regimes, including dynamic attack evolution and long-horizon progressive attacks.

Experimental Results

The evaluation of 12 models (including GPT-5.4, Claude Opus 4.6, Qwen3.6-Plus, and others) reveals significant gaps in current agent security:

Uneven Security Capabilities: Security is not a shared property of frontier models. Claude Opus 4.6 achieved the highest overall score (83.9), significantly outperforming the second-best model (Qwen3.6-Plus at 69.7). Many models, particularly lightweight ones like GLM-4.7-Flash, exhibited high unsafe outcome rates (UOR > 60%).
Decoupling of Safety and Awareness: A critical finding is the dissociation between preventing harm and recognizing attacks. Some models (e.g., Claude Sonnet 4.5 in IPI scenarios) achieved high Outcome Safety scores (>95) but very low Security Awareness scores (<25), suggesting they rely on conservative execution or task avoidance rather than genuine risk recognition.
Vulnerability to Long-Horizon and Stateful Attacks:
- Long-Horizon Progressive Attacks (LPA): These attacks, which unfold across multiple sessions, proved highly effective. Seven of twelve models had UOR > 50%, indicating a structural weakness in connecting dispersed risk signals across time.
- Memory Contamination (MC): Most models failed to treat persistent memory as untrustworthy, with UOR ranging from 3.8% (Claude Opus 4.6) to 67.3% (GLM-4.7-Flash).
- Skill Poisoning: Script-based camouflage (hiding malicious logic in executable scripts rather than manifests) caused a sharp collapse in safety scores across all models.
Impact of Attack Strengthening: Dynamic Attack Evolution (iterative refinement of prompts) significantly degraded the performance of most models, revealing that static benchmarks underestimate vulnerability to adaptive adversaries.

Significance and Claims

The paper claims that AgentCanary provides a systematic baseline for developing more reliable and secure agent systems. Its primary significance lies in shifting the focus of agent safety evaluation from "content compliance" (text generation) to "system integrity and action controllability" (real-world execution).

The authors emphasize that current agents often fail to recognize the attacks they face, particularly under conditions of compromised skills, persistent state, and long-horizon execution. The framework demonstrates that robust security requires the joint realization of preventing unsafe outcomes and explicitly recognizing adversarial intent. By providing a high-fidelity, trajectory-grounded evaluation pipeline, AgentCanary enables the diagnosis of specific failure modes (e.g., lack of cross-session reasoning, blind trust in memory) that are critical for the safe deployment of autonomous agents in high-privilege environments.