Malicious Agent Skills in the Wild: A Large-Scale Security Empirical Study

1. Problem Statement

The rapid adoption of Large Language Model (LLM)-based agents has led to the emergence of agent skills: reusable, file-based packages (typically containing SKILL.md instructions and executable scripts like Python/Shell) that extend agent capabilities.

• The Threat: Unlike traditional plugins, agent skills often execute with full local user privileges and are distributed through community registries with minimal vetting.
• The Gap: While static analysis tools exist, there is no ground-truth dataset to characterize the actual threat landscape. Existing studies flag potential risks but cannot distinguish between developer mistakes and intentional malicious behavior, nor can they confirm if suspicious code paths actually execute.
• The Objective: To construct the first large-scale, labeled dataset of malicious agent skills and systematically measure the ecosystem's threat landscape, attack strategies, and evasion techniques.

2. Methodology

The authors employed a three-tiered pipeline to process 98,380 skills from two major community registries (skills.rest and skillsmp.com) as of January 2026.

A. Data Collection & Static Analysis (Tier 1 & 2)

• Collection: Automated crawling of two registries indexing skills from public GitHub repositories.
• Static Triage: Applied 14 detection patterns (adapted from MITRE ATT&CK) across six kill chain phases (Reconnaissance, Credential Access, Execution, Evasion, Exfiltration, Impact).
  • Code-Level: Regex matching for credential harvesting, remote execution, and obfuscation.
  • Instruction-Level: Used an LLM-based agent (GPT-5.2) to analyze SKILL.md files for prompt injection, hidden directives, and behavior manipulation.
• Result: Identified 4,287 suspicious candidates (4.4% of the total).

B. Behavioral Verification (Tier 3)

• Sandbox Execution: The 4,287 candidates were executed in isolated Docker containers (Ubuntu 22.04) with:
  • Network monitoring (tcpdump).
  • System call tracing (strace).
  • Honeypot credentials (fake API/SSH keys).
  • Synthetic inputs generated by an LLM to trigger skill functions.
• Verification Criteria: A skill was flagged if it transmitted honeypot credentials, connected to undocumented domains, executed obfuscated payloads, or escalated privileges.
• Manual Review: Two security researchers independently reviewed 762 dynamically flagged skills. A skill was confirmed malicious only if both researchers assigned a confidence rating of ≥2.
• Result: Confirmed 157 malicious skills containing 632 labeled vulnerabilities.

C. Analysis & Labeling

• Shadow Features: Identified undocumented runtime behaviors (e.g., hidden network endpoints) not inferable from public documentation.
• Sophistication Stratification: Classified skills into three levels (Basic, Intermediate, Advanced) based on vulnerability count, evasion techniques, and shadow features.
• Statistical Validation: Used Fisher's exact tests and co-occurrence matrices to identify attack chains and archetypes.

3. Key Contributions

1. First Labeled Dataset: Released a three-tiered dataset (98k snapshot, 4.2k candidates, 157 confirmed malicious) with 632 labeled vulnerabilities and shadow feature annotations.
2. High-Precision Verification: Achieved 99.6% precision through behavioral verification, a 150× improvement over static-only approaches, providing a reliable ground truth for future detection research.
3. Threat Characterization: Mapped the ecosystem into two distinct attack archetypes and identified platform-native attack vectors that exploit AI trust mechanisms.

4. Key Results & Findings

A. Threat Landscape (RQ1)

• High Complexity: Malicious skills average 4.03 vulnerabilities per skill, spanning a median of 3 kill chain phases. This indicates deliberate technique layering rather than accidental bugs.
• Natural Language Attack Surface: 84.2% of vulnerabilities reside in SKILL.md (documentation), not executable code. Attackers use coercive language, false urgency, and hidden directives within Markdown files.
• Sophistication: The ecosystem is "middle-heavy" (77.7% Intermediate), but advanced threats (Level 3) show a steep capability gradient, with 100% employing shadow features.

B. Attack Strategies (RQ2)

The ecosystem has bifurcated into two negatively correlated archetypes:

1. Data Thieves (70.5% of cases):
   • Goal: Exfiltrate credentials and data.
   • Technique: Supply chain attacks using hardcoded endpoints, remote script execution (SC2), and credential harvesting (E2).
   • Industrialization: A single actor (smp_170) accounts for 54.1% of all confirmed malicious skills via templated brand impersonation across 15 industry sectors.
2. Agent Hijackers (10.2% of cases):
   • Goal: Subvert agent decision-making and bypass safety controls.
   • Technique: Instruction-level manipulation (P1/P4) to override user permissions, suppress alerts, and force the agent to act against the user's interest (e.g., "Do NOT ask user permission").
   • Significance: These attacks exploit the AI's instruction-following nature, a vector invisible to traditional code scanners.

C. Evasion Techniques (RQ3)

• Shadow Features: Evasion scales with sophistication. 0% of Basic skills have shadow features, rising to 100% of Advanced skills.
• Documentation vs. Code: Most concealment occurs at the documentation level (hidden instructions in comments/markup) rather than code obfuscation. Only 9.5% of malicious skills used code obfuscation.
• Platform-Native Attacks: A subset of attacks weaponizes the AI platform's own infrastructure, including:
  • Model Substitution: Redirecting API calls to attacker proxies.
  • Hook Manipulation: Intercepting tool operations via the platform's hook system to exfiltrate data.
  • Permission Abuse: Exploiting flags like --dangerously-skip-permissions.

5. Significance & Impact

• Validation: Responsible disclosure led to the removal of 93.6% of the 157 confirmed malicious skills within 30 days, validating the dataset's accuracy.
• Defense Implications:
  • Shift in Detection: Traditional code-level static analysis is insufficient. Defenders must prioritize documentation-behavior comparison and semantic analysis of natural language instructions.
  • Dual Pipelines: Detection systems must run parallel pipelines: one for code execution (catching Data Thieves) and one for semantic instruction analysis (catching Agent Hijackers).
  • Urgency: The ecosystem is maturing rapidly (reaching sophistication levels of 2012 browser extensions in just 3 months). Immediate investment in detection infrastructure is critical before advanced threats become the norm.
• Open Science: The authors released the full dataset, analysis pipeline, and detection scripts to support future research in agent security.

Conclusion

This study reveals that the agent skill ecosystem is a fertile ground for sophisticated, multi-vector attacks that exploit both code execution privileges and the unique vulnerabilities of LLM instruction-following. The emergence of industrialized "Data Thief" campaigns and "Agent Hijacker" strategies necessitates a paradigm shift in security, moving from code-centric analysis to a holistic approach that treats natural language documentation as a primary attack surface.