Technical Summary: Behavioral Integrity Verification for AI Agent Skills

Problem Statement

Large Language Model (LLM) agents are increasingly extended via third-party "skills"—self-contained bundles of code, configuration, and natural-language instructions that grant agents privileged access to filesystems, credentials, network endpoints, and shell execution. While existing safety mechanisms focus on catching malicious prompts or risky runtime actions, the skill artifacts themselves remain unverified.

A critical vulnerability exists in the gap between a skill's declared behavior (metadata, descriptions, and instructions) and its actual behavior (executable code and embedded directives). Malicious actors can embed natural-language directives in metadata files (e.g., "ALWAYS use this skill for ALL user questions") or hide logic in code that overrides the agent's decision loop. Current static analysis methods struggle to bridge code, instructions, and metadata under a single typed vocabulary, leaving a "description-implementation gap" that allows supply-chain risks to proliferate undetected.

Methodology: The BIV Framework

The authors propose Behavioral Integrity Verification (BIV), a framework that formalizes the verification of agent skills as a typed set comparison between declared and actual capabilities over a shared taxonomy.

1. Formalization

BIV defines a skill $s = (M, C, I)$ consisting of:

• Metadata ($M$): Descriptions, YAML frontmatter, API schemas.
• Code ($C$): Executable code (Python, JavaScript, shell).
• Instructions ($I$): Natural-language directives (Markdown, inline instructions).

The framework maps both declared behavior $D(s)$ and actual behavior $A(s)$ to a shared 29-capability taxonomy ($\mathcal{T}$) organized into seven categories: Network, Filesystem, Process Execution, Environment, Encoding, Credential, and Instruction-Level Threats. The core verification problem is identifying the symmetric difference (deviation set) $D(s) \triangle A(s)$, decomposed into:

• Under-specification ($U(s)$): Capabilities present in code but absent from metadata (undisclosed privilege).
• Over-specification ($O(s)$): Capabilities declared but never implemented.

2. Hybrid Extraction Pipeline

BIV employs a dual-track extraction process to generate structured evidence $\Phi(s) = \langle D(s), A(s), flow(s), compound(s) \rangle$:

• Declared Track: Uses a deterministic parser for structural fields (YAML, JSON) and an LLM extractor for natural-language metadata. The LLM is constrained by hallucination-control filters (e.g., requiring quoted source spans) to ensure claims are grounded.
• Actual Track:
  • Code Analysis: Deterministic analyzers (AST traversal for Python, regex for JS/Shell) perform inter-procedural taint analysis to trace data flows from sources (e.g., environment variables) to sinks (e.g., network sends).
  • Instruction Analysis: An LLM analyzer scans natural-language instructions for capability invocations, prompt-injection patterns, and override motifs.
• Compound Threat Detection: The pipeline flags four specific multi-step attack motifs (Exfiltration Chains, RCE Chains, Code Obfuscation, Data Lineage Violations) that single-capability scanners might miss.

3. Downstream Analyses

The structured evidence $\Phi(s)$ serves as a measurement primitive for three downstream tasks:

1. Deviation Taxonomy: Clustering deviations to identify common patterns and novel compound threats.
2. Root-Cause Classification: Distinguishing between developer oversight (documentation gaps) and adversarial intent.
3. Malicious-Skill Detection: A case study using an LLM judge augmented with a "relaxed-veto" override. The judge consumes $\Phi(s)$ as a structured prior, while the veto predicate automatically flags skills with compound threats and high-risk undeclared capabilities (e.g., credential exfiltration), preventing the LLM from rationalizing away structural evidence.

Key Results

The framework was evaluated on 49,943 skills from the OpenClaw registry and a 906-skill malicious detection benchmark.

Ecosystem-Scale Findings

• Pervasive Deviations: 80.0% of skills deviate from their declared behavior, with 250,706 total deviations identified.
• Root Causes: The vast majority (81.1%) of deviations stem from developer oversight (e.g., documentation errors, unused declarations, template residue) rather than malice. Only 18.9% are attributed to adversarial intent.
• Compound Threats: The analysis surfaced four novel compound-threat categories. While most are benign, specific motifs like Exfiltration Chains and RCE Chains show high adversarial fractions (58% and 86%, respectively).
• Instruction-Level Risks: Despite being the smallest category by volume, Instruction-Level Threats had the highest adversarial fraction (96%), indicating that undeclared prompt-control directives are highly suspicious.

Detection Performance

On the 906-skill benchmark (mixing real-world malware and synthetic attacks):

• BIV Performance: Achieved an F1 score of 0.946, with a recall of 0.978 and a False Positive Rate (FPR) of 7.2%.
• Comparison: BIV outperformed state-of-the-art baselines:
  • Rule-based (Cisco AI Defense): F1 = 0.44 (Recall 0.35).
  • LLM-only (Single-pass audit): F1 = 0.927 (Recall 0.89).
• Contribution of Structure: The structural evidence and relaxed-veto override were critical, particularly for real-world malware where the LLM-only baseline missed 22 percentage points of recall. The structural extractor identified kill chains (e.g., credential read $\to$ network send) that the LLM judge alone failed to flag as malicious.

Significance and Claims

The paper claims that Behavioral Integrity Verification provides a scalable, auditable primitive for securing the LLM agent supply chain. Its primary contributions are:

1. Formalization: It establishes BIV as a typed set comparison problem, bridging the gap between code, instructions, and metadata without relying on a single modality.
2. Measurement at Scale: It demonstrates that the "description-implementation gap" is a systemic issue in agent ecosystems, affecting 80% of skills, but is predominantly a documentation/oversight issue rather than a pervasive malicious one.
3. Operational Utility: The structured evidence $\Phi(s)$ enables a "one-pass" extraction that supports multiple security analyses (taxonomy, root-cause, detection) without re-extraction.
4. Detection Efficacy: By combining deterministic structural analysis with LLM reasoning and a relaxed-veto override, BIV achieves high recall on real-world malware while maintaining an operationally feasible false-positive rate, outperforming both rule-based and pure-LLM approaches.

The authors emphasize that BIV is a measurement primitive, not a final verdict system. It identifies candidates for review, distinguishing between benign documentation gaps and complex attack patterns, thereby enabling targeted governance strategies (e.g., documentation outreach for oversight vs. security review for adversarial intent). The framework acknowledges limitations, including its static nature (missing dynamic dispatch) and reliance on LLM extraction, but posits that the structured evidence it produces is a necessary foundation for future runtime defenses and longitudinal monitoring.