Malicious Or Not: Adding Repository Context to Agent Skill Classification

This paper presents the largest empirical security analysis of the AI agent skill ecosystem, demonstrating that incorporating repository context significantly reduces false positive malicious classifications from 46.8% to 0.52% while uncovering new attack vectors such as the hijacking of skills from abandoned GitHub repositories.

The Gist

Imagine you've just bought a brand new, super-smart robot assistant (like a digital butler) that can do anything for you: write code, check your emails, or manage your finances. But this robot is a bit limited out of the box. To make it truly useful, you need to give it "skills"—little digital tools or plugins, like adding a new app to your smartphone.

This paper is about a massive investigation into the safety of these robot skills.

Here is the story, broken down into simple parts:

1. The Problem: The "App Store" Panic

Imagine a world where people are selling these robot skills on different "App Stores." Recently, security guards (automated scanners) started checking these skills and screaming, "DANGER! MALICIOUS!"

In fact, some of these stores claimed that nearly half of all the skills were dangerous. It was like walking into a grocery store and being told that 50% of the apples are rotten. This caused a lot of panic. People started thinking, "Maybe I shouldn't use my robot assistant at all!"

2. The Investigation: The "Big Dig"

The researchers in this paper decided to investigate this panic. They didn't just look at the "app store" labels; they went on a massive scavenger hunt.

• The Scale: They collected 238,180 unique skills from three different marketplaces and from GitHub (a giant warehouse where developers store their code).
• The Old Way: They first looked at the skills in isolation, just like the security guards did. They asked, "Does this code look suspicious?"
• The Result: The old way still flagged a huge number of skills as dangerous.

3. The Twist: Context is King

Here is where the paper gets clever. The researchers realized that looking at a skill in isolation is like judging a book by its cover without reading the story.

The Analogy:
Imagine you find a knife in a kitchen.

• Scenario A: You find a knife lying alone in a dark alley. You think, "That's dangerous! Someone might get hurt!"
• Scenario B: You find the same knife in a chef's hand inside a busy, well-lit restaurant kitchen. You think, "That's fine. It's a tool for cooking."

The security scanners were only looking at the knife in the dark alley. They didn't see the kitchen.

The researchers developed a new method: Repository-Aware Analysis. They looked at the "kitchen" (the GitHub repository) where the skill lived. They asked:

• Is this skill part of a larger, trusted project?
• Does the code around it make sense?
• Is the developer an active, real person?

4. The Big Reveal: It Was Mostly a False Alarm

When they added this "context" to their analysis, the results changed dramatically.

• Before: The scanners said ~46% of skills were bad.
• After: With the "kitchen" context, they realized that 99.5% of those "bad" skills were actually just normal tools being misunderstood.
• The Real Danger: Only 0.52% of the skills were actually in truly malicious environments.

It turns out the security guards were crying wolf so often that they were scaring everyone away from perfectly safe tools.

5. The Real Villains: The "Abandoned Houses"

While most skills were safe, the researchers did find some real new dangers that nobody knew about yet.

The "Hijacked House" Attack:
Imagine you buy a house, but the previous owner moves away and forgets to change the locks. A stranger walks in, changes the address on the mailbox, and starts selling you "fresh bread" (which is actually poison).

• The researchers found that some skills point to abandoned GitHub repositories.
• Because the original owners left, bad actors could take over those empty digital "houses," change the code, and hijack the skills.
• They found 121 skills that were vulnerable to this. One of these "hijacked" skills had been installed over 1,000 times!

6. The Leaky Faucet

They also found a privacy leak. One of the marketplaces (ClawHub) was accidentally showing the private email addresses of the people who created the skills. It's like a store displaying the home addresses of all its customers on the front window.

Summary: What Should We Take Away?

1. Don't Panic: The ecosystem isn't 50% dangerous. It's actually quite safe (99.5% benign) if you look at the whole picture.
2. Look at the Whole Picture: You can't judge a tool just by its description. You need to see where it lives and who built it.
3. Fix the Locks: We need to stop using "abandoned" digital houses for our tools. If a project is dead, the marketplace needs to move the skill to a new, safe home so hackers can't sneak in.

In short: The paper tells us that the "wolf" isn't as scary as we thought, but we do need to make sure we aren't leaving our digital front doors unlocked.

Technical Summary

1. Problem Statement

The rapid adoption of autonomous AI agents (e.g., Claude Code, OpenClaw) has led to the emergence of "skill" ecosystems—modular extensions that allow agents to perform tasks like API access, code execution, and data retrieval. These skills are distributed via dedicated marketplaces (ClawHub, Skills.sh, SkillsDirectory) and GitHub repositories.

The Core Issue: Current security scanners used by these marketplaces report alarmingly high rates of malicious skills (ranging from 6% to 46.8%). However, these scanners typically analyze skills in isolation, focusing solely on the SKILL.md description and immediate code without considering the broader context of the hosting repository. This isolationist approach leads to a high rate of false positives, potentially stifling the ecosystem's growth and eroding user trust. Furthermore, the reliance on external GitHub repositories for distribution introduces supply chain risks, such as repository hijacking, which are currently undocumented.

2. Methodology

The authors conducted the largest empirical security analysis of the AI agent skill ecosystem to date, utilizing a three-stage measurement pipeline:

A. Cross-Platform Data Collection

• Scale: Collected 238,180 unique skills from three major marketplaces (ClawHub, Skills.sh, SkillsDirectory) and GitHub.
• Technique: Used APIs for marketplaces and GHArchive snapshots for GitHub to bypass search result limits. They deduplicated skills using SHA-256 hashing.
• Static Analysis: Enumerated files to identify programming languages, extracted endpoints (URLs/IPs), and scanned for embedded secrets (credentials) using a modified TruffleHog tool.

B. Malicious Classification (Baseline)

• Comparative Analysis: Compared results from existing marketplace scanners (VirusTotal, Gen Agent Trust Hub, Snyk, Socket) against two independent tools:
  1. Cisco Skill Scanner: An open-source tool using static analysis, bytecode integrity checks, and AST-based dataflow analysis.
  2. LLM-based Feature Extraction: A local GPT-5.3 (Codex) instance analyzing the SKILL.md and executable logic against a 25-feature questionnaire to detect behavioral indicators (e.g., data exfiltration, persistence).

C. Repository-Aware Analysis (The Novel Contribution)

To address the high false-positive rate, the authors developed a Repository-Aware Analysis framework. Instead of judging a skill in isolation, they evaluated the skill's congruence with its surrounding GitHub repository.

• Codebase Alignment Score (70% weight): Uses LLMs to assess if the repository's domain, source code, and documentation align with the skill's description.
• Metadata Score (30% weight): Evaluates repository maturity (age, stars, forks, activity, size) to distinguish established projects from newly created, low-credibility repos.
• Final Verdict: A weighted "Repository Context Score" determines if a flagged skill is likely benign within its specific context.

3. Key Contributions

1. Largest Ecosystem Dataset: Creation of a dataset containing 238,180 unique skills, providing a baseline for future longitudinal studies.
2. Repository-Aware Scoring: A novel methodology that integrates repository context (codebase and metadata) into security classification, significantly reducing false positives.
3. Discovery of New Attack Vectors:
   • Repository Hijacking: Identified 121 skills pointing to 7 abandoned GitHub repositories that are vulnerable to hijacking (where an attacker can recreate the repo under the same name).
   • Metadata Leakage: Discovered that the ClawHub API inadvertently exposes the email addresses associated with users' GitHub accounts, which are not visible on public profiles.

4. Key Results

A. Inconsistency in Current Scanners

• High Variance: Malicious classification rates varied wildly across scanners, from 3.79% (Socket) to 41.93% (OpenClaw Scanner).
• Low Consensus: Among 27,111 skills analyzed by five different scanners, only 33 skills (0.12%) were flagged as malicious by all scanners.
• Aggregation Bias: When aggregating skill-level flags to the repository level, the "malicious" rate for repositories jumped to 45.89% (vs. ~19% for individual skills), suggesting that popular repositories are often unfairly labeled malicious due to a single flagged skill within them.

B. Impact of Repository Context

• Drastic Reduction in False Positives: When applying the repository-aware analysis to the 2,887 skills previously flagged as malicious by the Cisco scanner and LLM:
  • Only 0.52% (15 skills) remained in repositories deemed suspicious.
  • 96% of flagged skills were found to be embedded in repositories where the documentation and codebase aligned with the skill's functionality.
• Context Sensitivity: The same skill could receive significantly different scores depending on the repository it was hosted in, proving that context is a critical factor in risk assessment.

C. Ecosystem Observations

• Secrets: Only 12 functional credentials were found across the entire dataset, likely because developers are more aware of public code visibility on GitHub compared to mobile apps.
• Hijacking Risk: 121 skills were identified as vulnerable to hijacking via abandoned repositories, with one skill having over 1,000 recorded installations.

5. Significance and Implications

• Re-evaluating Risk: The paper argues that the current perception of the AI agent skill ecosystem as highly dangerous is largely an artifact of isolationist scanning. By adding repository context, the actual risk surface is revealed to be much smaller (0.52% vs. 46.8%).
• Operational Guidance: The findings suggest that marketplace operators should move away from purely static or description-based scanning toward context-aware analysis to reduce false positives and maintain ecosystem health.
• Supply Chain Security: The discovery of repository hijacking vectors highlights a critical supply chain vulnerability in the current "index-only" distribution model (where marketplaces link to GitHub). The authors recommend a shift toward direct distribution models (hosting skills directly) to mitigate hijacking risks.
• Privacy: The exposure of user emails via the ClawHub API represents a significant privacy leak that needs immediate remediation.

In conclusion, this study provides a robust framework for distinguishing between genuinely malicious AI skills and benign extensions that merely appear suspicious when analyzed without context, offering a path toward a more secure and trustworthy AI agent ecosystem.