💻 computer science

MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills

This paper introduces MalSkillBench, the first runtime-verified benchmark comprising 3,944 malicious AI agent skills, to demonstrate that current detection tools fail to jointly analyze code and instructions, necessitating a new approach that reasons across task intent, code, and prompts to secure the agent supply chain.

Original authors: Wenbo Guo, Wei Zeng, Chengwei Liu, Xiaojun Jia, Yijia Xu, Lei Tang, Yong Fang, Yang Liu

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Wenbo Guo, Wei Zeng, Chengwei Liu, Xiaojun Jia, Yijia Xu, Lei Tang, Yong Fang, Yang Liu

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've hired a super-smart robot assistant to help you write software. To make this robot even more powerful, you let it download "skills" from the internet—like little apps or plugins that tell the robot how to do specific tasks, like "calculate taxes" or "format this document."

These skills are a bit like a recipe card (instructions written in plain English) glued to a kitchen appliance (actual code that runs).

The Problem: The "Poisoned Recipe"

The researchers found that hackers are starting to poison these skills. They create fake skills that look helpful but contain hidden traps.

The Code Trap: The "kitchen appliance" part might secretly steal your passwords or delete your files.
The Instruction Trap: The "recipe card" part might trick the robot into ignoring its safety rules or doing something it shouldn't, like sending your private data to a stranger.

The scary part is that these two traps often work together. A skill might look innocent on the recipe card but have a dangerous machine attached, or vice versa.

The Big Gap: We Didn't Have a Good Test

Until now, security experts trying to build "skill detectors" (like antivirus software for these robot skills) were flying blind.

They didn't have a big enough list of bad skills to test against.
The few bad skills they had were mostly one type (like fake "crypto" scams), so detectors got really good at spotting those but terrible at spotting anything else.
It was like testing a fire alarm only on smoke from burning toast, then claiming it will catch a fire from a grease explosion.

The Solution: MalSkillBench (The "Stress Test")

The authors built MalSkillBench, the first giant, verified testing ground for these malicious skills.

How they built it:

The Factory: They used an AI to generate thousands of fake malicious skills.
The Sandbox: They didn't just guess if a skill was bad. They actually ran the skill inside a safe, isolated digital cage (a "sandbox").
The Proof: If the skill actually did something bad (like tried to steal a file or trick the robot) while running in the cage, it got a "verified" label. If it failed to do the bad thing, they sent it back to the factory to try again.

They ended up with 3,944 verified bad skills and 4,000 good skills, covering 108 different types of attacks.

What They Discovered

When they tested current security tools against this new, massive test, the results were surprising:

Code is easy to catch, words are hard: Detectors are great at spotting bad code (like a virus in a file) but terrible at spotting bad instructions (like a trick in the recipe). It's easier to see a bomb than to see a lie.
The "Wild" Data is a Lie: If you only test detectors on the few bad skills found "in the wild" (on the internet), you get the wrong answer. One security tool looked like a hero on the wild data, but when tested on the full MalSkillBench, it was actually one of the worst. It was like a doctor who only treats colds and thinks they are a genius at curing everything, until you give them a patient with a broken leg.
Old Tools Don't Fit: You can't just use tools designed for regular software (to catch bad code) or tools designed for chatbots (to catch bad prompts). A malicious skill is a hybrid. Using a tool for just one half of the problem leaves the other half wide open.
The Real Danger: The most dangerous skills aren't just stealing files; they are tricking the robot's brain. Some skills try to rewrite the robot's identity, change its goals, or make it ignore its safety rules. Current tools are mostly blind to these "mind-hacking" attacks.

The Bottom Line

To keep our AI assistants safe, we can't just look at the code or just read the instructions. We have to understand the relationship between them: Does this instruction make sense for this task, or is it a trick?

The paper provides the first real "stress test" to help developers build better detectors that can spot these hybrid tricks, rather than just guessing based on old, incomplete data. They have made all their data and tools public so others can help solve this puzzle.

Technical Summary: MalSkillBench

Problem Statement

AI coding agents (e.g., Claude Code, Gemini CLI) increasingly rely on third-party "skills"—distributable packages bundling natural-language instructions, executable scripts, and tool permissions. These skills introduce a novel hybrid attack surface where risk resides not purely in code nor purely in prompts, but in the semantic relationship between the two. Current detection efforts face three critical gaps:

Lack of Ground Truth: Existing datasets are either withheld by industry (e.g., Snyk, Antiy) or are too small (e.g., 157 samples in prior academic work) to systematically evaluate detectors.
Narrow Attack Coverage: Available "in-the-wild" samples are heavily skewed toward a single attack pattern (dependency impersonation leading to malware delivery), leaving instruction-layer attacks and agent-control hijacks empirically invisible.
Fragmented Evaluation: Detectors are evaluated on private datasets with disparate metrics, preventing fair comparison across Code Injection (CI), Prompt Injection (PI), and Mixed (MIXED) vectors.

Consequently, the community lacks a reliable understanding of which detection tools generalize and which merely overfit to specific, narrow attack patterns.

Methodology

The authors present MalSkillBench, the first runtime-verified benchmark for malicious agent skills. The construction relies on a closed-loop Generate-Verify-Feedback pipeline designed to create high-quality, verified ground truth.

1. Attack Taxonomy and Knowledge Base

The benchmark defines a three-dimensional taxonomy spanning 108 attack cells:

Attack Vector ( $v$ ): Code Injection (CI), Prompt Injection (PI), and Mixed (MIXED).
Malicious Behavior ( $b$ ): 15 categories, including traditional supply-chain behaviors (B1–B9, e.g., data exfiltration, reverse shells) and agent-native behaviors (B10–B15, e.g., role hijacking, safety bypass, instruction override).
Insertion Strategy ( $s$ ): Specific methods for payload placement (e.g., new script files, function injection, steganography, full camouflage).

The pipeline seeds generation from two knowledge bases: IntelliGraph (3,026 malicious PyPI packages) for CI patterns and WildJailbreak/CCS'24 corpora for PI patterns. These are mapped to the taxonomy using LLM-based labeling.

2. Generation and Verification Loop

Generation Agent: Uses a modified Qwen3.5-35B (abliterated to remove refusal mechanisms) to synthesize candidate skills. It retrieves attack examples, selects a benign "camouflage" template from a pool of 3,458 legitimate skills, and injects the malicious payload according to the target taxonomy cell.
Verification Agent: Executes candidates inside a Docker sandbox instrumented with strace and inotifywait. It employs a two-layer verification process:
- Layer 1 (Deterministic): Matches system-call traces (network, file, process) against declared Indicators of Compromise (IOCs).
- Layer 2 (Semantic): Uses an LLM (GPT-5.4-mini) to analyze agent text outputs and trace summaries to confirm behavioral intent, particularly for PI attacks which leave no syscall traces.
Feedback: If a candidate fails verification, the agent receives structured feedback (reason, observed behaviors, suggestions) and retries up to three times. Only samples where the malicious behavior is confirmed (runtime yield) enter the dataset.

3. Dataset Composition

The final benchmark ( $\mathcal{D}$ ) comprises:

3,214 Generated Skills: Verified via the closed-loop pipeline, ensuring balanced coverage across all 108 taxonomy cells.
703 Wild Skills: Manually collected and verified from public registries (e.g., ClawHub), representing real-world attacks.
27 Test Samples: Curated from existing tool test suites.
4,000 Benign Skills: High-confidence, widely downloaded skills used to measure false positives.

Key Contributions

MalSkillBench Dataset: A runtime-verified benchmark containing 3,944 malicious and 4,000 benign skills, labeled across a 108-cell taxonomy.
Closed-Loop Pipeline: A novel automated framework that migrates real-world attack patterns into the skill format and validates them via sandbox execution, turning label claims into behavioral evidence.
Systematic Evaluation: The first unified evaluation of 12 detection tools (9 skill-specific, 5 supply-chain scanners, 6 prompt-injection defenders) under identical conditions.

Empirical Results

RQ1: Attack Realizability

Realizability is uneven across the attack surface.

Code Injection (CI): 94.5% verification yield.
Mixed (MIXED): 91.9% yield.
Prompt Injection (PI): 75.8% yield.
Insight: PI-only reasoning behaviors (e.g., Instruction Override, Goal Hijacking) are the hardest to realize (63–70% yield), indicating a structural fragility in generating instruction-only attacks that successfully alter agent behavior.

RQ2: Wild Analysis

The "in-the-wild" attack surface is strikingly narrow.

Dominance: 86.3% of wild skills use dependency impersonation (fake prerequisites), and 86.6% result in Malware Delivery.
Concentration: 81% of wild samples originate from just two accounts, largely driven by a single cryptocurrency-theft campaign.
Agent-Native Threats: A small tail (1.7%) targets the agent control plane (session lifecycle, identity, instruction hierarchy) rather than the host, a vector absent in traditional package ecosystems.

RQ3: Skill-Specific Detection

No current detector is uniformly reliable.

Performance: The strongest tool (Sentry Skill Scanner, full mode) achieves 98.4% recall on the full benchmark but drops significantly on PI and agent-control attacks.
The "Wild" Bias: Evaluating solely on wild data is systematically misleading. For example, VirusTotal ranks near the bottom on the full benchmark (21.6% recall) but jumps to the top (87.9% recall) when evaluated only on wild data—a 66-point swing. This occurs because wild data is dominated by dependency impersonation, which signature-based tools detect well, while ignoring instruction-layer attacks.

RQ4: Tool Transferability

Existing single-domain tools fail to transfer effectively.

Supply-Chain Scanners: High recall (e.g., OSSGadget at 97.3%) comes with massive false positives (up to 3,979 on 4,000 benign samples) because they flag benign dependencies as malicious.
Prompt-Injection Defenses: Similarly, they either over-trigger on code-heavy skills or miss operational attacks that lack jailbreak-like phrasing.
Combination Failure: Naive combinations (OR/AND logic) of these tools do not recover the missing semantic relationship between code and instructions. OR combinations inherit high false positives; AND combinations collapse recall (often <6%).

Significance and Claims

The paper claims that detecting malicious skills requires joint reasoning over task intent, code, and instructions, rather than inspecting them separately. The primary significance of MalSkillBench is that it exposes the limitations of current evaluation methodologies:

Measurement Distortion: Relying on "wild-only" data or narrow subsets artificially inflates the performance of tools that happen to match the dominant (but narrow) attack patterns, while obscuring failures on instruction-level attacks.
Hybrid Nature: The risk of a skill is not a localized artifact but a semantic relationship (e.g., "Is this dependency justified by the task?"). Current tools, which read either code or prompts in isolation, cannot capture this relationship.
Future Direction: The benchmark provides the necessary ground truth to develop detectors that can reason across the hybrid code-instruction space, addressing the "blind spots" in current security practices.

The authors emphasize that their work does not invent new attack vectors but systematically measures the realizability and detectability of existing threats within the specific context of AI agent skills, revealing that the instruction layer remains a significant and under-protected vulnerability.