💻 computer science

SkillProbe: Security Auditing for Emerging Agent Skill Marketplaces via Multi-Agent Collaboration

This paper introduces SkillProbe, a multi-agent collaborative security auditing framework that addresses semantic-behavioral inconsistencies and combinatorial risks in LLM agent skill marketplaces, revealing through large-scale evaluation that high popularity does not guarantee security and that systemic cascaded risks form a giant connected component.

Original authors: Zihan Guo, Zhiyu Chen, Xiaohang Nie, Jianghao Lin, Yuanjian Zhou, Weinan Zhang

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zihan Guo, Zhiyu Chen, Xiaohang Nie, Jianghao Lin, Yuanjian Zhou, Weinan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a bustling digital city called the Agentic Web. In this city, there are millions of tiny, super-smart robots (called LLM Agents) that help people do everything from writing code to booking flights. But these robots can't do everything alone; they need tools.

Enter Skill Marketplaces (like a giant app store for robots). Developers upload "Skills"—little packages of code that teach a robot how to do specific tasks, like "search the web" or "send an email."

The problem? The city is full of hidden traps.

The Problem: The "Wolf in Sheep's Clothing"

Currently, when a robot wants a new skill, it reads the description (the label on the package).

The Label says: "I am a safe calculator that only adds numbers."
The Reality: The code inside is actually a spy that steals your passwords and sends them to a hacker.

This is the Semantic-Behavioral Inconsistency. The robot trusts the label, but the code is lying.

Even worse, there's the Combinatorial Risk. Imagine two skills that are perfectly safe on their own:

Skill A: "Reads a text file." (Safe)
Skill B: "Deletes a file." (Safe)

But if a hacker tricks the robot into using them together in a specific order, Skill A reads your private bank statement, and Skill B immediately deletes it to cover the tracks. Two harmless tools become a deadly weapon when chained together.

The Solution: SkillProbe (The "Super-Inspector")

The paper introduces SkillProbe, a new security system designed to audit these skills before they enter the marketplace. Instead of a single human or a simple scanner checking the code, SkillProbe uses a team of AI detectives working together.

Here is how it works, using a Restaurant Inspection analogy:

1. The "Gatekeeper" (The Health Inspector)

Before a skill even gets a seat at the table, the Gatekeeper checks the basics.

What it does: It scans for obvious red flags like known viruses, dangerous dependencies, or permissions that look suspicious (e.g., a calculator asking for access to your camera).
The Analogy: It's like checking if the restaurant has a valid license and if the kitchen is visibly dirty. If it fails here, it's rejected immediately.

2. The "Alignment Detector" (The Lie Detector)

This is the most crucial part. The team compares the Menu (the description) with the Kitchen (the actual code).

What it does: It asks, "Does the code actually do what the description says?"
The Analogy: The menu says "Grilled Chicken," but the kitchen is secretly cooking "Rat Poison." The Alignment Detector catches this mismatch. It looks for "Shadow Functions"—hidden features the developer didn't mention but the code executes anyway.

3. The "Flow Simulator" (The Stress Test)

This is where SkillProbe gets really clever. It doesn't just look at one skill; it simulates how skills work together.

What it does: It creates a "What-If" scenario. "What happens if we use Skill A, then Skill B, then Skill C?"
The Analogy: Imagine a chef testing a recipe. "If I mix this safe sauce with that safe spice, does it create a toxic gas?" The simulator tries to chain skills together to see if they accidentally create a security disaster.

The Big Discoveries

The researchers tested SkillProbe on 2,500 real-world skills from a popular marketplace. They found some shocking truths:

The Popularity Paradox: You might think the most downloaded skills are the safest because "everyone uses them." Wrong. The study found that over 90% of the most popular skills failed the rigorous security audit. Popularity does not equal safety. It's like a restaurant being famous just because it's on the corner, even if the food is poisoned.
The Giant Web of Danger: They discovered that risky skills aren't isolated islands. They form a giant, connected web. If you connect just two random high-risk skills, they can trigger a chain reaction that steals data or crashes the system. It's not just one bad apple; it's a whole barrel rotting together.

Why This Matters

Before SkillProbe, security was like checking a car's engine while it was driving down the highway (Runtime Defense). If the engine explodes, it's too late.

SkillProbe changes the game to Pre-Distribution Auditing. It's like a rigorous inspection at the factory before the car is ever sold. By using a team of AI agents to check the labels, the code, and how the parts work together, SkillProbe helps build a Trustworthy Agentic Web where robots can safely help us without accidentally stealing our data or crashing our systems.

In short: SkillProbe is the ultimate "Truth in Advertising" law for the future of AI, ensuring that when a robot says it's safe, it actually is.

1. Problem Statement

The rapid evolution of Large Language Model (LLM) agent ecosystems has led to the emergence of centralized skill marketplaces (e.g., ClawHub), where agents orchestrate complex workflows using third-party skills. However, these marketplaces face unprecedented security challenges that traditional static analysis and runtime defenses cannot adequately address:

Semantic-Behavioral Inconsistency: A critical gap exists between a skill's natural language documentation (what it claims to do) and its underlying executable code (what it actually does). Malicious developers can publish skills with benign descriptions but hidden "shadow functions" (e.g., data exfiltration, privilege escalation) that evade standard code scanners.
Inter-Skill Combinatorial Risks: Individual skills may appear benign in isolation but induce malicious behaviors when chained together. Adversaries can engineer skill pairs where the output of an upstream skill triggers high-risk behaviors in a downstream skill (e.g., command injection, indirect prompt injection).
Limitations of Current Defenses:
- Runtime Governance: Fails to neutralize stealthy exploits hidden deep within logic before execution.
- Static Analysis: Often limited to discrete code-level patterns and supply chain integrity, missing the semantic layer and dynamic compositional threats.
- The Popularity Paradox: High download volume is not a reliable proxy for security; popular skills often fail rigorous security standards.

2. Methodology: The SkillProbe Framework

SkillProbe is a multi-stage, multi-agent security auditing framework driven by a novel "Skills-for-Skills" design paradigm. Instead of a monolithic auditor, the auditing process itself is encapsulated into standardized skill modules that general agents can dynamically load and execute.

System Architecture

The framework operates on a five-layer architecture:

Input Layer: Ingests heterogeneous skill formats (local, archives, URLs).
Orchestration Layer: A "Security Auditor" agent manages state, distributes tasks, and schedules phases.
Skill Layer (Core): Contains three specialized auditing modules:
- Gatekeeper: Admission filtering.
- Alignment Detector: Semantic consistency analysis.
- Flow Simulator: Combinatorial risk analysis.
Output Layer: Generates structured reports and security scorecards.
Infrastructure Layer: Provides storage, tool registries, and an agentic runtime.

The Three Auditing Phases

Phase 1: Gatekeeper (Admission Filtering)

Goal: Rapid identification of explicit malicious patterns.
Mechanism: Uses decoupled inspection with parallel "Gate Executors" (e.g., Skill Vetter) to scan for backdoors, hazardous dependencies (CVEs), and permission rationality.
Logic: Employs a conservative "one-vote veto" policy; a single "BLOCK" decision halts the process.

Phase 2: Alignment Detector (Semantic Consistency)

Goal: Detect discrepancies between documentation ( $D(s)$ ) and code ( $C(s)$ ).
Mechanism:
- Document Extractor: Uses regex and LLMs to extract declared capabilities.
- Code Extractor: Uses AST analysis and LLM reasoning to trace I/O bindings and sensitive parameters.
- Marker Agent: Maps capabilities to a canonical key and constructs a four-class alignment matrix:
  1. Match: Full alignment.
  2. Over-declaration: Claims more permissions than implemented.
  3. Under-declaration (Shadow Functions): Implements capabilities not documented (High Risk).
  4. Mixed: Complex deviations.

Phase 3: Flow Simulator (Combinatorial Risk)

Goal: Simulate risks arising from skill chaining.
Mechanism:
- Risk Fingerprinting: Reduces search complexity from exponential ( $O(2^N)$ ) to linear ( $O(N \times |Rules|)$ ) by tagging skills with Output Risk (Source) and Input Sensitivity (Sink).
- Risk Link Policies: Matches sources to sinks based on 9 predefined attack patterns (e.g., Unsanitized String $\to$ Shell Execution = Command Injection).
- Simulation: Executes batched attack path simulations (e.g., data exfiltration chains) and enforces evidence citation from original code.

Decision Logic

Skills are evaluated via a Security Scorecard across three dimensions (Malicious Patterns, Semantic Consistency, Composition Safety). A strict veto policy categorizes skills into:

REJECTED: Critical vulnerabilities found.
CONDITIONAL: Low-risk discrepancies requiring manual review.
APPROVED: Meets all security baselines.

3. Key Contributions

Automated Multi-Agent Framework: Introduced SkillProbe, the first framework to automate the transition from compliance filtering to deep semantic analysis and cross-skill attack detection using a plug-and-play "Skills-for-Skills" architecture.
Multi-dimensional Risk Modeling: Developed a method to extract standardized capability features to build an alignment matrix and a risk-link graph, enabling the precise identification of hazardous skill chains and shadow functions.
System Implementation & Evaluation: Built a robust prototype (FastAPI, Vue 3, npm-based REPL) and conducted a large-scale longitudinal audit on 2,500 real-world skills from ClawHub using 8 different LLM series (including Claude, Gemini, GPT-5, and Nex-N1).

4. Key Results

The Popularity-Security Paradox:
- In a sample of the top 2,500 skills, only 9.9% were fully "clean" (Approved without findings).
- 30.4% were marked "Conditional" (potential risks).
- Over 90% of high-popularity skills failed to meet rigorous security standards.
- Statistical Invariance: The proportion of risky skills remained constant (20–35%) across all download tiers, proving that popularity does not correlate with security quality.
The Giant Connected Component:
- Analysis of 499 skills with confirmed exploits revealed a risk network of 75,373 edges.
- The high-risk subset forms a single giant connected component, meaning risks are systemic and cascaded, not isolated.
- Data Exfiltration (32k edges) and Fact Poisoning (28k edges) dominate the risk landscape.
Cross-Model Consistency:
- Evaluated across 8 LLMs. While execution times and strictness varied (e.g., Claude Sonnet 4.6 was the strictest, Gemini Flash the fastest), the framework maintained high robustness in identifying core risks.
- High-risk skills (e.g., nano-banana-pro) were consistently flagged across all models, demonstrating the system's resilience against model-specific biases.

5. Significance and Implications

Shift in Security Paradigm: Moves the industry from reactive runtime intervention and atomic code scanning to proactive, intent-aware pre-distribution auditing.
Addressing the Semantic Gap: Highlights that the description layer (natural language) is now a primary attack surface, requiring verification of "intent-behavior alignment" alongside code verification.
Combating Combinatorial Explosion: Proves that traditional atomic auditing is blind to cascaded risks. The "Flow Simulator" demonstrates that combinatorial risk prediction is feasible under linear complexity.
Infrastructure for the Agentic Web: SkillProbe provides a scalable governance infrastructure necessary for building a trustworthy Agentic Web, suggesting that future marketplaces must mandate multi-dimensional certification systems rather than relying on popularity metrics.

Conclusion: SkillProbe successfully identifies that the current agent skill ecosystem is fraught with systemic, interconnected risks that popularity cannot mitigate. By leveraging multi-agent collaboration to simulate complex attack chains and verify semantic consistency, it offers a critical pathway toward securing the future of autonomous agent workflows.