🤖 AI

Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems

The paper introduces Proteus, a self-evolving red-team framework that demonstrates current skill vetting systems significantly underestimate security risks by revealing how adaptive, feedback-driven attackers can iteratively bypass audits to create lethal agent skill variants.

Original authors: Zhaojiacheng Zhou

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhaojiacheng Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where you can download "skills" for your AI assistant, just like you download apps on your phone. These skills tell the AI how to do specific tasks, like checking your bank account or managing your calendar. But what if a hacker creates a "poisoned" skill that looks harmless but secretly steals your data?

This paper introduces Proteus, a digital security researcher designed to test how good our current safety guards are at catching these poisoned skills.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Whac-A-Mole" Game

Currently, when someone submits a new skill, it goes through a Security Auditor (a digital guard). The guard checks the skill's code and its description (documentation) to see if it's dangerous.

The Flaw: Most security tests only check the skill once. If the guard says "Pass," the skill is installed.
The Reality: A smart hacker doesn't just submit one bad skill and give up. They watch why the guard rejected it, tweak the skill slightly, and try again. They keep doing this until they find a version that slips past the guard but still does the damage.

The paper calls this "Adaptive Leakage." It's the risk that a hacker can keep trying until they break the lock.

2. The Solution: Proteus (The Self-Evolving Hacker)

The researchers built Proteus, an AI agent that acts like a relentless, learning hacker. Instead of just trying to trick the guard once, Proteus plays a multi-round game:

Submit: It sends a skill to the guard.
Listen: If the guard says "No," Proteus reads the reason (e.g., "This code looks suspicious" or "The description doesn't match the code").
Evolve: Proteus uses that feedback to rewrite the skill. It might change the code, rewrite the description, or rearrange how the skill connects to other tools.
Repeat: It keeps doing this until it finds a version that passes the guard and successfully performs the harmful action (like stealing data) in a safe, simulated sandbox.

3. The Five Axes of Attack

Proteus doesn't just change one thing; it attacks the skill from five different angles simultaneously, like a master lockpicker trying different tools:

The Goal: What is the skill trying to do?
The Structure: How is the skill organized?
The Code: The actual instructions.
The Channel: How it talks to the outside world.
The Documentation: The written description that explains what the skill does.

Proteus can mix and match changes in all these areas to confuse the guard.

4. The "Round Contract" (The Testing Arena)

To make sure Proteus is actually finding real holes, the researchers built a strict testing arena with three steps for every attempt:

The Audit: Does the guard let it through?
The Sandbox: Does the skill actually run without crashing?
The Oracle: Did it actually do the harmful thing (like deleting a file or stealing a password)?

Proteus only counts as a "win" if it passes all three steps.

5. What They Found

The researchers tested Proteus against two real-world security guards (one open-source and one from a major tech company) and two different AI models.

The Results: Proteus was terrifyingly effective.
- In just 5 rounds of trying, it successfully bypassed the guards 40% to 90% of the time.
- It didn't just find one trick; it found 438 different ways to break the system.
- Even the strongest guard they tested (AI-Infra-Guard) failed to stop Proteus about 41% of the time.
- The "Transfer" Trick: When Proteus learned how to break one guard, it could often use those same tricks to break the other guard without needing to learn anything new.

6. The Big Takeaway

The paper concludes that our current way of checking AI skills is underestimating the danger. We are checking skills like a security guard checking a single photo of a person. But a real attacker is like a shapeshifter who keeps changing their face until the guard lets them in.

Proteus proves that if we don't build systems that can handle attackers who learn and adapt, our AI ecosystems will have huge, hidden holes that hackers can easily exploit.

In short: We thought our security guards were good at spotting bad skills. This paper shows that if a bad actor is smart enough to learn from their mistakes, those guards are actually quite easy to fool.

Technical Summary: Proteus – A Self-Evolving Red Team for Agent Skill Ecosystems

1. Problem Formulation: Adaptive Leakage

The paper addresses a critical gap in the security evaluation of Large Language Model (LLM) agent skills. While existing research has identified risks in static, hand-crafted malicious skills, it largely fails to measure the residual risk remaining when an attacker iteratively revises a skill using feedback from auditors and runtime environments.

The authors define Adaptive Leakage as the deployment risk where a budgeted attacker can iteratively revise a third-party skill until it simultaneously:

Passes the static or LLM-based audit.
Executes successfully in the target runtime.
Produces verified, oracle-detectable harm.

Current evaluations treat auditors as one-shot classifiers against fixed inputs. However, because open-source auditors (e.g., SkillVetter, AI-Infra-Guard) often publish their rule sets and findings, attackers can locally reproduce audit results. This creates a "grey-box" channel where an attacker can use structured audit findings and runtime traces to refine their attack across multiple rounds, a capability not captured by single-shot red-teaming or scalar-feedback baselines.

2. Methodology: The Proteus Framework

Proteus is a grey-box, self-evolving red-team framework designed to measure adaptive leakage. It operates through a closed-loop system comprising four coupled components:

A. The Five-Axis Skill-Attack Space

Proteus decomposes a skill variant $s$ into a 5-tuple $(o, \tau, c, \kappa, d)$ :

$o$ (Objective): The attack goal (e.g., data exfiltration, privilege escalation).
$\tau$ (Topology): The skill chain structure (single skill or $k$ -chain).
$c$ (Code): The executable artifact.
$\kappa$ (Side-effect Channel): The mechanism for harm (e.g., file write, network egress).
$d$ (Documentation): The context-setting documentation loaded into the agent's prompt.

B. The Grey-Box Round Contract

Each evaluation round follows a defender-agnostic pipeline: Audit $\to$ Sandbox $\to$ Oracle.

Audit: The skill is submitted to an auditor (e.g., SkillVetter or AI-Infra-Guard).
Sandbox: If the audit passes, the skill executes in a containerized environment (OpenClaw).
Oracle: A rule-based judge verifies if the execution produced the targeted side-effect (lethality).
Feedback Quadruple: The system returns a structured tuple $\langle b, \ell, F, R \rangle$ $⟨ b, ℓ, F, R ⟩$ :
- $b$ : Boolean audit verdict (bypassed?).
- $\ell$ : Boolean lethality verdict (harm achieved?).
- $F$ : Structured audit findings (reason codes, severities, evidence pointers).
- $R$ : Runtime trace/advice (e.g., which sensors were triggered or blocked).

C. Self-Evolving Mutator

The red-team agent is an LLM-based tool-use loop that consumes the feedback quadruple as in-context side-information (a "verbal gradient") rather than a scalar reward. It utilizes a catalog of 12 mutation operators (e.g., purpose-coherent-embed, code-offload, thematic-chain-insert) to modify the five axes of the skill.

D. Three-Layer Evolution Process

Proteus employs a nested search strategy:

Layer 1 (Free Discovery): Searches freely within the parent objective to find the first successful bypass ( $b \land \ell$ ).
Layer 2 (Path Expansion): Upon success, generates alternative implementations of the same attack objective (e.g., swapping cron for systemd timers) without consuming the main round budget.
Layer 3 (Surface Expansion): Transfers learned implementation patterns (structural motifs) to new attack objectives, testing if the attacker can generalize beyond the initial seed.

3. Experimental Setup

Auditors: SkillVetter (open-source, MIT-0) and AI-Infra-Guard (Tencent, Apache-2.0).
Targets: GPT-5.4-mini and GLM-5.
Mutators: DeepSeek-V4-Pro and Kimi-2.6.
Seeds: 20 malicious seed skills generated via graph-guided chain composition over a 47k-node skill graph (ClawHub).
Budget: $T=5$ rounds for Phase-1; off-budget expansion for Phase-2.

4. Key Results

Phase-1: Iterative Bypass

Across eight configuration cells, Proteus achieved an Attack Success Rate (ASR@5) of 40–90%.

Learning Curves: Proteus demonstrated positive learning-curve slopes (LCS), indicating that success rates increased significantly with each round of feedback.
Baseline Comparison:
- Blackbox Baseline: When provided only with a scalar pass/fail verdict (no structured findings), the success rate plateaued at ~43% with negligible learning slope.
- PyRIT: A standard external red-teaming tool (narrative rewrite only) achieved only 5/20 successful seeds, highlighting the necessity of mutating code and topology, not just documentation.
Auditor Performance: The stronger auditor, AI-Infra-Guard, reduced success rates by 10–30 percentage points compared to SkillVetter, but Proteus still achieved up to 41.3% success against it.

Phase-2: Path and Surface Expansion

After finding a successful variant, Proteus expanded the attack surface:

Total Variants: Generated 438 jointly bypassing and lethal variants.
SkillVetter: Bypassed at $\ge 93\%$ in every expansion cell.
AI-Infra-Guard: Still admitted 25–41% of variants.
Surface Expansion: The agent successfully transferred attack patterns to new objectives (e.g., moving from persistence to data exfiltration) using shared structural motifs, demonstrating emergent generalization.

Cross-Auditor Transfer

A critical finding is the transferability of learned attacks. 87.7% of variants evolved against the stronger AI-Infra-Guard also bypassed SkillVetter without any new mutation. This suggests that adaptive leakage is not auditor-specific but represents a fundamental vulnerability in the skill vetting paradigm.

5. Significance and Claims

The paper claims that current skill vetting substantially underestimates residual risk when evaluated against adaptive, feedback-driven attackers.

Measurement Gap: Existing security studies treat auditors as static classifiers. Proteus demonstrates that when an attacker has access to structured feedback (the "grey-box" channel), they can iteratively refine attacks to bypass both static analysis and LLM-based narrative checks.
The "Verbal Gradient": The study empirically validates that using structured audit findings as in-context side-information is significantly more effective than scalar rewards. Collapsing feedback to a binary verdict reduces the learning slope by an order of magnitude.
Dual-Channel Vulnerability: The results highlight that the combination of code execution and context-setting documentation creates a dual attack surface. Successful attacks often require coordinated mutations across multiple axes (e.g., rewriting the narrative to match the code while offloading the payload to a binary).
Ecosystem Risk: The high rate of cross-auditor transfer implies that vulnerabilities discovered against one auditor likely persist across others, suggesting a systemic risk in the agent skill ecosystem that cannot be solved by simply swapping auditors.

The authors conclude that adaptive leakage is a measurable, significant threat that requires new defense mechanisms capable of handling iterative, feedback-driven adversaries, rather than relying on one-shot static audits.