Technical Summary: SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction

Problem Statement

Agent skills—reusable packages bundling natural language instructions, reference materials, and executable scripts—occupy a privileged position in agent workflows. Unlike untrusted external context (e.g., web documents), agents are implicitly expected to follow skill instructions, consult references, and execute scripts. This implicit trust renders third-party skills a vulnerable attack surface. Malicious content embedded in a skill can be interpreted as legitimate guidance and executed with the agent's full privileges over files, shell commands, and credentials.

Existing research on skill-based attacks suffers from two primary limitations:

1. Lifecycle Blindness: Most benchmarks evaluate poisoned skills within a single task execution, ignoring the persistence of skills across sessions. They fail to capture attacks where an initial benign execution silently mutates skill content, deferring harm to a subsequent reuse.
2. Ad-hoc Risk Evaluation: Current studies often rely on benchmark-specific risk lists rather than a systematic taxonomy of skill-relevant risks, leading to incomplete coverage of potential attack outcomes.

Methodology

1. Attack Scenarios and Risk Taxonomy

The authors introduce SkillHarm, a benchmark designed to evaluate agent vulnerabilities across the skill-use lifecycle. It defines two distinct attack scenarios:

• Fixed-Payload Poisoning (FPP): A poisoned skill package contains a fixed malicious payload that triggers a harmful outcome immediately upon invocation during a single task execution.
• Self-Mutating Poisoning (SMP): The skill contains a mutation hook. An initial task execution appears benign but silently modifies persistent skill content. The harmful outcome is deferred, materializing only when a later task reuses the compromised package.

To ensure systematic coverage, SkillHarm categorizes risks into 12 types organized by the workflow component targeted:

• Data Pipeline Exploitation: Data exfiltration, output manipulation, and data poisoning.
• System Environment Exploitation: Privilege escalation, unauthorized file modification, backdoor injection, denial of service, malware deployment, and system corruption.
• Agent Autonomy Exploitation: Goal hijacking, anti-forensics, and proxy attacks.

2. Automated Construction: AutoSkillHarm

Given the high heterogeneity of skills, risks, and attack scenarios, manual design is impractical. The authors developed AutoSkillHarm, an automated pipeline driven by coding agents using natural-language harnesses. The pipeline operates in three stages:

1. Attack Target Selection: Identifies reachable skill files (for FPP) or task pairs sharing skills (for SMP) based on exposure rates in benign baseline trajectories.
2. Attack Design: A coding agent (primarily Claude Code) instantiates abstract risk types into concrete adversarial goals. It materializes payloads by modifying skill files while maintaining plausible workflow connections. For SMP, it implements deferred-mutation mechanisms (e.g., exit-triggered callbacks). The agent also generates deterministic evaluators to verify harmful end states.
3. Quality Filtering: Candidates undergo iterative self-refinement and LLM-based detection to ensure stealth. A reviewer agent then performs trajectory-grounded analysis to discard invalid or non-executable candidates, retaining only sound attacks.

Key Results

The benchmark comprises 879 attack samples across 71 skills and 12 risk types. The authors evaluated six model-harness configurations (including Claude Code, Codex, Gemini CLI, and OpenCode) against these attacks.

• High Vulnerability: Current agents remain highly vulnerable.
  • FPP: Attack Success Rates (ASR) reached up to 86.3% (Codex GPT-5.4).
  • SMP: ASR reached 69.3%, despite the stricter requirement of successful mutation and cross-session reuse.
• Latent Risk (Conditional ASR): A significant gap exists between ASR and Conditional ASR (cASR), which measures success given the agent engages with the poisoned file. For example, in SMP, cASR for Claude Opus 4.7 jumped from 9.4% to 41.5% (+32.1%) when conditioned on engagement. This indicates that many apparent "failures" are due to agents not accessing the targeted file, not genuine resistance.
• Refusal Behavior: Explicit refusal is rare. Only Claude-family models showed noticeable refusal behavior, and even then, rates dropped significantly in SMP scenarios (e.g., from 25.3% in FPP to 2.1% in SMP for Sonnet 4.6), suggesting temporal decoupling obscures malicious intent.
• Ineffectiveness of Current Defenses:
  • Skill Scanners: Contextualized injections frequently evade detection. The strongest scanner configuration detected only 55.6% of FPP and 68.8% of SMP injections.
  • Defensive System Prompts: Adding warnings to system prompts reduced ASR in some cases but remained unreliable, with nearly all configurations staying above 70% ASR.
• Task Utility: Poisoned skills generally preserved the original task success rate, meaning users receive plausible outputs while the agent silently executes harmful side effects.

Significance and Claims

The paper claims that SkillHarm establishes skill-based attacks as a distinct and under-examined region of the agent-security landscape. Its primary contributions are:

1. Lifecycle Awareness: It is the first benchmark to jointly evaluate direct single-session compromise and deferred cross-session compromise, revealing that agents remain vulnerable even when skills are reused across sessions.
2. Systematic Risk Coverage: By organizing 12 risk types into a workflow-grounded taxonomy, it provides a comprehensive view of potential harms beyond ad-hoc lists.
3. Automated Scalability: The AutoSkillHarm pipeline demonstrates that complex, heterogeneous attacks can be constructed automatically at scale using natural-language harnesses, overcoming the limitations of manual or template-based generation.
4. Security Implications: The findings reveal that current safety mechanisms (scanners, prompts, and model training) are insufficient against skill-based attacks. The paper argues that effective mitigation likely requires mechanisms beyond prompt-level interventions, such as runtime guardrails or safety post-training that specifically targets the recognition of malicious intent within skill content.

The authors release the benchmark, dataset, and construction pipeline to support the development of defenses that address the inherent trust agents place in skill content.