💻 computer science

Structured Security Auditing and Robustness Enhancement for Untrusted Agent Skills

This paper introduces SkillGuard-Robust, a framework that transforms pre-load auditing of untrusted Agent Skills into a robust three-way classification task, achieving near-perfect malicious-risk recall and attack consistency across diverse package evaluations while highlighting ongoing challenges in external-source transfer.

Original authors: Lijia Lv, Xuehai Tang, Jie Wen, Jizhong Han, Songlin Hu

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lijia Lv, Xuehai Tang, Jie Wen, Jizhong Han, Songlin Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new assistant for a complex job. Instead of just reading a single resume (a "prompt"), this assistant arrives with a whole toolkit: a manual (SKILL.md), a set of scripts, reference documents, and a history of how they were built (repository context).

The problem is that a bad actor could hide a dangerous instruction inside the reference manual or a script, disguised as a normal tool. If you only read the resume, you miss the trap. If you just dump all the files into a pile of text and ask a smart AI to "read it all," the AI might get confused by the sheer volume or tricked by a clever rewrite of the instructions.

This paper introduces SKILLGUARD-ROBUST, a new security system designed to inspect these "toolkits" before they are allowed to work. Here is how it works, using simple analogies:

The Problem: The "Flat" vs. "Structured" View

The Old Way (Flattening): Imagine taking a complex machine, smashing it into a pile of scrap metal, and asking a guard to find a hidden bomb in the pile. The guard might miss the bomb because the parts aren't connected in a way that makes sense. In the paper, this is called "flattening the package." It loses the structure, so hidden attacks (like a hidden override in a reference file) get missed.
The New Way (Structured): SKILLGUARD-ROBUST doesn't smash the machine. It keeps the parts organized. It knows which file is the manual, which is the script, and which is the history. It looks for cross-file clues—like a script that says "call this remote helper," which is only explained in the reference manual.

The Three Main Tricks Attackers Use

The paper identifies three specific ways attackers hide in these toolkits:

The Hidden Override: Like a sticky note on a manual that says, "Ignore the safety instructions on page 1."
The Disguised Transfer: Like a delivery truck labeled "Backup" that is actually smuggling contraband.
The Remote Bootstrap: Like a "setup wizard" that secretly installs a backdoor instead of just the software.

How SKILLGUARD-ROBUST Works (The Four-Stage Process)

Instead of asking one super-smart AI to make a final decision instantly, this system uses a four-stage assembly line to catch mistakes that other systems miss.

Stage 1: The Detective (Structured Evidence Extraction)
The system first organizes the files. It doesn't just read them; it maps out who talks to whom. It asks: "Does this script rely on that reference file?" It builds a map of the toolkit's structure so no hidden connections are missed.

Stage 2: The Specialist (Selective Semantic Verification)
Most toolkits are clearly safe or clearly dangerous. But some are "fuzzy"—they look suspicious but might be innocent.

The Innovation: The system doesn't waste time asking a powerful AI to review every toolkit. It only sends the "fuzzy" ones to a specialist AI.
The Analogy: Imagine a security checkpoint. If your bag looks normal, you walk through. If it looks weird, a specialist opens it and inspects the contents closely. This saves time and focuses power where it's needed.

Stage 3: The Judge (Conflict-Aware Chain Arbitration)
Sometimes, a toolkit has two conflicting signals: one part looks like a "remote setup" (suspicious) and another looks like a "data transfer" (dangerous).

The Problem: A simple AI might just say "It's risky" and stop there, or get confused about which risk is the real one.
The Fix: This stage acts like a judge who weighs the evidence. It asks: "Is this a harmless setup, or is it a disguised data theft?" It makes a final call on which "chain of events" is the dominant one.

Stage 4: The Consistency Checker (Anchor-Consistency Consolidation)
Attackers often try to trick the system by rewriting the instructions slightly (e.g., changing "steal data" to "move files") while keeping the same bad intent.

The Fix: The system looks at the original version of the toolkit and its rewritten versions together. If the rewrites are clearly dangerous, but the original was labeled "suspicious," the system corrects the original label to match the truth. It ensures the system doesn't get tricked just because the words changed.

The Results: Why It Matters

The authors tested this system on hundreds of toolkits, including some that were never seen before (like new, real-world open-source projects).

The "Strong Baseline" (The Smart AI): Even the smartest existing AI models (like Qwen2.5-14B) were good at spotting some risk, but they often failed to correctly identify the most dangerous ones. They would say, "This looks suspicious," but miss that it was actually a confirmed attack.
SKILLGUARD-ROBUST: By using the four-stage process, the system achieved near-perfect scores (around 97-99% accuracy) in identifying dangerous toolkits and, crucially, in recognizing that a rewritten attack was still an attack.

The Bottom Line

The paper concludes that to secure these "Agent Skills," you can't just rely on a bigger, smarter AI to read everything at once. You need a structured process that:

Respects the file organization.
Only uses heavy AI power on the confusing cases.
Resolves conflicts between different types of risks.
Checks for consistency when attackers try to rewrite their tricks.

The paper admits that while this works great on known and "frozen" sets of data, the real world is still a challenge, and the system isn't a magic bullet for every possible future attack. But for the specific problem of auditing these toolkits before they run, this structured approach is a massive improvement over current methods.

1. Problem Statement

The rise of Agent Skills (packaged capabilities containing SKILL.md, scripts, reference documents, and repository context) introduces a critical security vulnerability: third-party untrusted content is directly integrated into the agent's execution chain.

The Core Challenge: Existing security guardrails typically treat auditing as a single-prompt filtering task, often flattening the entire package into one text span. This approach fails because:
1. Distributed Evidence: Risk indicators (e.g., hidden override instructions, disguised data transfers) are distributed across multiple files and file roles, not concentrated in a single description.
2. Label Instability: Under semantics-preserving rewrites (attacks that change surface text but keep intent), current models suffer from "label drift," inconsistently classifying malicious packages as merely "suspicious" or "benign."
3. Collapse of Distinction: Strong remote baselines often flag risk but fail to stably recover the specific "malicious" class, collapsing the distinction between suspicious (needs review) and malicious (needs blocking).

The paper reframes the task as a robust three-way classification problem (Benign, Suspicious, Malicious) that requires cross-file evidence aggregation and stability against rewrite attacks.

2. Methodology: SKILLGUARD-ROBUST

The authors propose SKILLGUARD-ROBUST, an error-decomposition induced architecture. Instead of training a single end-to-end black-box classifier, the system uses a staged, interpretable pipeline designed to resolve specific ambiguities found in failure analysis.

The framework consists of four stages:

Stage I: Structured Evidence Extraction

Goal: Recover the package structure rather than flattening it.
Mechanism: Instead of treating the package as one text span, the system lifts each file $(x_i, r_i)$ into a role-conditioned evidence vector. It assigns weights to different file roles (e.g., SKILL.md, script, reference) and aggregates risk signals using a noisy-or function.
Benefit: This preserves strong evidence from critical single files while accumulating weak signals dispersed across the package, preventing the "max-over-files" rule from obscuring distributed risks.

Stage II: Uncertainty Triggering and Semantic Verification

Goal: Resolve local semantic overlaps (e.g., distinguishing benign synchronization from disguised transfer).
Mechanism: A selective semantic verifier is invoked only when the initial extraction yields uncertain scores (triggered by specific thresholds on risk signals).
Benefit: This avoids the cost of running a heavy verifier on every sample. It focuses computational resources on the "hard" cases where surface forms are ambiguous, outputting chain-level judgments (Override, Transfer, Bootstrap) rather than a final package label.

Stage III: Conflict-Aware Chain Arbitration

Goal: Resolve conflicts between competing semantic chains (e.g., Remote Bootstrap vs. Disguised Transfer).
Mechanism: The system fuses structured evidence scores with semantic verification outputs. It introduces a conflict-gating variable to determine if a sample is dominated by a malicious transfer chain or a remote bootstrap chain.
Benefit: This stage explicitly distinguishes between cases where a "transfer" is actually a "bootstrap" (which might be policy-dependent), repairing errors where single-shot judges fail to decide which chain is decisive.

Stage IV: Anchor-Consistency Consolidation

Goal: Ensure stability across rewrite variants (Attack Exact Consistency).
Mechanism: The system analyzes clusters of an "anchor" sample and its "rewrite" variants. If rewrites are consistently classified as "malicious" but the anchor is "suspicious," and specific confidence thresholds are met, the anchor is promoted to "malicious."
Benefit: This repairs residual inconsistencies where the semantic meaning is stable across rewrites, but the initial adjudication lagged behind.

3. Key Contributions

Formalization: Defined pre-load security auditing as a package-level decision problem, identifying three recurring semantic attack chains: Hidden Override, Disguised Transfer, and Remote Bootstrap.
Diagnostic Insight: Demonstrated that the primary failure mode of strong baselines is not "risk blindness" but the instability of the suspicious/malicious distinction under rewrites and cross-file conflicts.
Architecture: Introduced SKILLGUARD-ROBUST, a factorized auditing pipeline that combines structure recovery, selective verification, and consistency-preserving adjudication.
Benchmarking: Constructed SKILLGUARDBENCH and augmented it with public-ecosystem extensions, creating five large evaluation views (ranging from 254 to 404 packages) to test performance across main distributions, frozen stress, boundary cases, and external ecosystem transfers.

4. Experimental Results

The method was evaluated on five large views (Main, All-HO, Int.-stress, Boundary, Ext.-eco) against strong baselines like Qwen2.5-14B (full-package) and BUNDLEJUDGE.

Performance on Held-Out Aggregate (404 packages):
- Overall Exact Accuracy: 97.30%
- Risk Malicious Recall: 98.33%
- Attack Exact Consistency: 98.89%
Performance on External Ecosystem (254 packages):
- Overall Exact Accuracy: 99.66%
- Risk Malicious Recall: 100.00%
- Attack Exact Consistency: 100.00%

Key Findings:

Baselines Fail on Stability: Strong baselines (like Qwen2.5-14B) often flag risk (high "Flagged Accuracy") but collapse the "malicious" class into "suspicious," leading to a large gap between flagged accuracy and malicious recall.
Factorization Works: The staged approach significantly outperforms both flat single-shot judges and structured single-shot judges (like BUNDLEJUDGE), proving that structured evidence aggregation and consistency repair are necessary.
Cost Efficiency: The method incurs low remote costs because the semantic verifier is only triggered for uncertain samples, unlike baselines that re-read every package end-to-end.

5. Significance and Conclusion

This paper establishes that robust skill auditing is a structural decision problem, not merely a text classification task.

Shift in Paradigm: It moves away from "larger models solve everything" to "structured pipelines solve specific failure modes." The results show that preserving package structure, localizing uncertainty, and repairing decision residuals are more effective than simply scaling up model size.
Practical Impact: The proposed method provides a deployable, interpretable framework for pre-loading security checks in agent systems, significantly reducing the risk of malicious third-party skills entering the execution chain.
Limitations & Future Work: While the method achieves near-saturation on frozen and public-ecosystem views, the paper acknowledges that "open-world transfer" (mixed-language, deeply nested, or adaptive attacker rewrites) remains an open challenge. Future work should focus on multi-verifier cross-checking and handling more complex source shifts.

In summary, SKILLGUARD-ROBUST demonstrates that by decomposing the auditing task into structured extraction, targeted verification, and consistency repair, it is possible to achieve high robustness and stability in securing untrusted agent skills against sophisticated, multi-file attacks.