Technical Summary: Agent Skill Security: Threat Models, Attacks, Defenses, and Evaluation

1. Problem Statement

The rapid evolution of Large Language Model (LLM) agents from conversational interfaces to autonomous systems capable of executing complex, multi-step objectives has driven the adoption of reusable agent skills. These skills encapsulate tool invocations, execution policies, and semantic metadata, enabling agents to interact dynamically with external environments. However, this shift fundamentally alters the security assumptions of LLM agents. Unlike traditional prompt-based interactions, agent behavior is now influenced by externally authored executable artifacts whose metadata, permissions, and workflows may evolve independently of the underlying language model.

Existing security research has primarily focused on prompt injection, jailbreak attacks, and runtime execution vulnerabilities. These efforts typically analyze individual attack vectors in isolation, lacking a unified framework to reason about security across the complete lifecycle of reusable agent skills. Consequently, there is no systematic methodology for identifying trust boundaries, categorizing lifecycle-specific threats, or evaluating security mechanisms under a common experimental setting. This fragmentation obscures how vulnerabilities propagate across stages and which security assumptions are violated during agent execution.

2. Methodology: The SkillSec-Eval Framework

To address these gaps, the authors present SkillSec-Eval, a lifecycle-aware evaluation framework designed to systematically assess the security of reusable agent skills. Rather than treating an LLM agent as a monolithic pipeline, SkillSec-Eval decomposes the skill ecosystem into explicit operational boundaries.

2.1 The Skill Lifecycle and Threat Taxonomy

The framework models the skill lifecycle as six sequential stages, each representing a distinct trust boundary:

1. Authoring: Creation of skills (workflow definitions, metadata, permissions).
2. Storage: Publication to repositories (preservation of artifacts, provenance, versioning).
3. Retrieval: Semantic search to identify candidate skills relevant to user requests.
4. Selection/Planner: Reasoning over metadata to select the optimal workflow.
5. Execution Runtime: Invoking tools and interacting with external resources.
6. Evolution: Continuous updates, version changes, and dependency modifications.

Based on this lifecycle, the authors develop a threat taxonomy (Figure 1, Table 1) spanning five categories of attacks:

• Repository Admission: Structural attacks (missing signatures, tampered payloads, version rollback) and semantic attacks (permission inflation, metadata deception, hidden prompt injection).
• Retrieval: Attacks manipulating ranking algorithms (keyword stuffing, semantic camouflage, Sybil publication, trigger poisoning).
• Selection: Attacks manipulating planner decisions via metadata (planner deception, fake recommendations, misleading descriptions, permission deception).
• Execution: Runtime behaviors (unsafe composition, unauthorized tool invocation, information-flow violations, data exfiltration).
• Evolution: Exploiting accumulated trust via updates (permission escalation, instruction injection, tool substitution, dependency compromise, publisher compromise).

2.2 Experimental Setup

The framework was instantiated using a controlled benchmark consisting of 327 real-world benign skills from the SkillMCP repository, organized into 15 capability clusters (e.g., DevOps, cloud operations, security). The evaluation involved:

• Attack Generation: Creating adversarial variants targeting specific lifecycle boundaries (e.g., injecting malicious metadata while preserving executable workflows for planner attacks).
• Defense Implementation:
  • Admission: A hybrid controller combining rule-based structural validation with LLM-based semantic intent verification.
  • Retrieval: A validation layer performing semantic diversity filtering, metadata consistency checks, and permission justification.
  • Planner: A metadata sanitization layer removing prompt injection patterns and normalizing fabricated trust indicators.
  • Execution: A sandboxed runtime with dynamic taint tracking and policy enforcement.
  • Evolution: A continuous validation pipeline treating updates as new admission events.
• Metrics: Evaluation used stage-specific metrics such as Malicious Admission Rate (MAR), Attack Success Rate (ASR), nDCG@5, and Policy Violation Detection Rate.

3. Key Contributions

The paper makes four primary contributions:

1. Lifecycle-Oriented Threat Taxonomy: A systematic organization of security threats based on the trust boundaries violated throughout the skill lifecycle, connecting traditionally separate research areas like software supply-chain security and semantic retrieval attacks.
2. SkillSec-Eval Framework: A modular evaluation framework enabling reproducible security assessment across repository admission, retrieval, planning, execution, and evolution.
3. Comprehensive Benchmark: A dataset of enterprise-scale skill repositories, capability-clustered workloads, and standardized attack scenarios.
4. Empirical Study: A comprehensive evaluation of representative attacks across the entire lifecycle, providing insights into the security properties of modern reusable skill ecosystems.

4. Key Results

The empirical evaluation yielded the following findings:

• Repository Admission: Structural validation alone reduced the Malicious Admission Rate (MAR) to 52.9% but failed against semantic intent manipulation. A hybrid defense (structural + LLM semantic validation) lowered MAR to 7.9%, though it introduced a 20% False Positive Rate (FPR), highlighting the trade-off between security and developer friction.
• Semantic Retrieval: Undefended retrieval was highly vulnerable; Sybil attacks resulted in an average of 2.84 malicious clones in the Top-5 results (93.20% ASR). The proposed Retrieval Validation Layer reduced ASR for Keyword Stuffing and Semantic Camouflage to below 10% and Sybil attacks to 26.59%, though vector-space density attacks (Trigger Poisoning) remained partially successful.
• Planner Selection: Planners were highly susceptible to social-proof cues, with "Fake Recommendations" achieving a 45.64% baseline attack success rate. Metadata validation eliminated "Misleading Description" attacks entirely and reduced "Fake Recommendation" attacks by over 80%.
• Runtime Execution: In the absence of monitoring, 100% of execution attacks succeeded. The proposed defense blocked 87.0% of unauthorized operations but achieved only 66.67% Taint Propagation Accuracy. The study identified that semantic taint evasion (e.g., LLM paraphrasing sensitive data) remains a significant challenge for traditional dynamic taint tracking.
• Skill Evolution: Continuous validation successfully intercepted 92.5% of malicious updates (MDR). However, this strictness resulted in a 37.0% FPR, where legitimate developer refactoring was flagged as suspicious, underscoring the difficulty of maintaining robust zero-day resistance without impacting legitimate development.

5. Significance and Claims

The paper claims that vulnerabilities arise at multiple lifecycle stages beyond execution, challenging the prevailing focus on prompt injection and runtime safety. The authors assert that:

• Reusable skills introduce attack surfaces spanning both natural-language reasoning and conventional software execution.
• No single defense is sufficient; a defense-in-depth strategy covering the entire lifecycle is necessary because advanced attacks can bypass individual semantic or runtime filters.
• Traditional software supply-chain defenses (e.g., signatures, SBOMs) are fundamentally blind to the semantic layer of LLM-driven execution and cannot reason about metadata influence on retrieval or planner intent.

The authors conclude that SkillSec-Eval provides a foundational framework for future research into secure agent ecosystems, emphasizing that lifecycle-aware security analysis is critical for the trustworthy deployment of reusable agent skills. The paper explicitly notes limitations, including the lack of longitudinal data for trust decay and the exclusion of attacks originating from the underlying LLM itself (e.g., model jailbreaks), positioning these as complementary areas for future investigation.