Technical Summary: SkillVetBench

Problem Statement

Open agentic platforms (e.g., OpenClaw, NanoClaw) enable community contributors to publish reusable "skills" that agents invoke at runtime. While this extensibility drives ecosystem growth, it introduces a critical supply-chain risk: malicious contributors can embed harmful behavior within skills that appear benign under superficial inspection. Recent incidents, such as the ClawHavoc campaign which injected 1,184 malicious skills into the ClawHub marketplace, demonstrate that existing defenses are insufficient.

The core challenge identified is that current security vetting tools lack a systematic benchmark to evaluate both malicious-skill detection and runtime verification. Traditional static scanners fail to detect threats encoded in natural-language instructions (e.g., prompt injection, memory poisoning) or those that only manifest through cross-component interactions at runtime. Conversely, dynamic analysis without semantic grounding produces uninterpretable logs. There is a gap in the ability to verify whether a skill's static claims align with its actual executable behavior.

Methodology

The authors present SkillVetBench, a two-stage security vetting benchmark framework designed to evaluate open agentic skill ecosystems. The framework is built upon a dataset of 100 skills (78 confirmed malicious, 22 benign controls) drawn from the live OpenClaw repository, including samples from the ClawHavoc campaign.

Stage 1: Semantic Analysis (LLM-as-a-Judge)

The first stage performs a static analysis of the complete skill artifact (natural-language specifications, executable code, configuration files, and tool interfaces).

• Mechanism: An LLM acts as a "judge" to reason about declared intent, cross-component attack structures, and compositional risks.
• Evaluation Dimensions: Skills are scored on five agentic risk dimensions (0–3 scale):
  1. Instruction Fidelity Risk (IFR): Susceptibility to prompt injection or instruction override.
  2. Data Gravity (DG): Sensitivity of data accessed (e.g., credentials, private keys).
  3. Action Irreversibility (AI): Whether effects can be undone (e.g., deletions vs. reads).
  4. Blast Radius (BR): Scope of harm (single user vs. cross-platform).
  5. Chain Amplification (CA): Potential danger when composed with other skills.
• Output: These scores are aggregated into a Skill Agentic Risk Score (SARS) to classify skills as Benign, Suspicious, or Malicious. This stage specifically targets threats invisible to signature-based scanners, such as those hidden in natural-language instructions.

Stage 2: Programmatic Analysis (Sandbox Execution)

The second stage executes skills flagged as "Suspicious" or "Malicious" in Stage 1 within an instrumented, isolated Docker sandbox.

• Environment: A local OpenClaw stack is deployed with network access, filesystem operations, command execution, and credential access monitored.
• Process: The sandbox issues both benign prompts (to test declared functionality) and adversarial prompts (to probe injection, privilege abuse, and data exfiltration).
• Verification: A skill is escalated from "Suspicious" to "Malicious" only if the execution log contains a concrete, attributable trace of harmful behavior (e.g., filesystem modification, outbound network communication, or unauthorized command execution).
• Evidence: This stage generates auditable evidence traces (261,891 gateway log lines in the study) to confirm runtime threats.

Key Contributions

1. SkillVetBench Framework: The first two-stage benchmark that couples LLM-based semantic analysis with controlled runtime execution to vet agentic skills. It bridges the gap between static intent analysis and dynamic behavior verification.
2. Comprehensive Vulnerability Taxonomy: The benchmark defines and evaluates seven specific vulnerability categories relevant to agentic systems:
   • Code-execution: Command Injection, Unsafe File Operations.
   • Instruction-layer/Agentic: Prompt Injection, Memory Poisoning, Privilege Abuse.
   • Data/Supply Chain: Data Exposure, Supply Chain risks.
3. Novel Metrics and Scoring: Introduction of the SARS metric and the integration of CVSS v4.0 for static scoring, allowing for a multi-dimensional risk assessment that accounts for agentic-specific threats like instruction hijacking and chain amplification.
4. Empirical Dataset: A curated benchmark of 78 confirmed malicious skills and 22 benign controls, providing a ground-truth dataset for evaluating future security tools.

Experimental Results

The authors evaluated SkillVetBench against eight baselines, including official marketplace scanners (ClawScan, ClawVet), static analysis tools (CodeBERT), and LLM-based approaches (zero/few-shot prompting, SkillProbe, SkillSieve).

• Detection Performance: SkillVetBench achieved zero false negatives across all 78 confirmed malicious skills, outperforming all baselines.
  • Baselines missed up to 89% of malicious skills, particularly in categories lacking code-level signatures (e.g., Prompt Injection, Memory Poisoning). For instance, ClawScan missed 84% of Prompt Injection skills, and VirusTotal missed 100% of them.
  • SkillVetBench was the only system to issue "Malicious" verdicts backed by sandbox execution traces for 5 Command Injection and 1 Memory Poisoning skill.
• Runtime Behavior Analysis: Analysis of 261,891 log lines revealed that confirmed attacks are concentrated in a small set of high-permission primitives: exec, write_file, install_skill, spawn, and subagent. Read-only tools produced nearly zero confirmed attacks.
• Static vs. Dynamic Discrepancy: The study found that while static scanners (and CVSS v4.0) effectively identify direct code exploits, they fail to capture compositional and instruction-layer threats. For example, Data Exposure and Supply Chain risks were often rated "Low" by static CVSS scoring but "Suspicious" by the multi-dimensional SARS framework due to their runtime context.
• Evaluator Sensitivity: The study showed that LLM judge performance varies significantly by model architecture and size. Models with at least 7B parameters and strong instruction-following alignment (e.g., Qwen2.5-32B, Llama-3.1-7B) were necessary for reliable detection, while smaller or less-aligned models exhibited systematic under-detection or over-sensitive, poorly-calibrated scoring.

Significance and Claims

The paper claims that SkillVetBench addresses a critical evaluation gap in open agentic ecosystems. Its primary significance lies in demonstrating that reliable skill vetting requires both semantic threat detection and runtime verification.

• Complementarity: The authors position SkillVetBench not as a replacement for existing tools, but as a complementary framework that addresses the structural blind spots of current systems: the inability to reason over instruction-layer threats and the absence of runtime verification.
• Evidence-Based Verdicts: By grounding final verdicts in concrete execution traces rather than static suspicion alone, the framework provides a reproducible pipeline for assessing whether a skill's behavior is triggered under realistic conditions.
• Supply Chain Security: The work highlights that the "openness" of agentic platforms creates a unique attack surface where malicious behavior is often distributed across natural-language instructions, configuration files, and executable code, necessitating a holistic, two-stage defense strategy.

The authors conclude that without such systematic benchmarks and evidence-producing vetting systems, open agentic platforms remain vulnerable to supply-chain attacks that evade traditional static analysis.