🤖 machine learning

SkillConsist: Detecting Inconsistencies in Agent Skills via Bidirectional Graph Alignment

SkillConsist is a novel framework that detects inconsistencies between declared and implemented agent skills by constructing bidirectional behavior graphs and performing graph alignment and differencing, achieving state-of-the-art performance on a newly constructed 633-skill benchmark.

Original authors: Chaofan Meng, Yuhang Zheng, Yingnan Zhou, Sihan Xu

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chaofan Meng, Yuhang Zheng, Yingnan Zhou, Sihan Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your computer doesn't just follow orders but acts like a helpful assistant, a digital butler that can browse the web, manage your files, or even book your vacation. To do this, it uses "Agent Skills"—pre-made tools or recipes that tell the computer exactly how to perform a specific task. Think of these skills like instructions in a cookbook: the "declaration" is the delicious description on the menu promising a "Spicy Tacos" dish, while the "implementation" is the actual cooking happening in the kitchen.

The problem arises when the menu lies. Maybe the chef (the code) accidentally adds a secret ingredient that makes the tacos toxic, or perhaps they forgot to add the salsa entirely, even though the menu promised it. In the digital world, these mismatches are dangerous. If a computer agent picks a skill based on a false promise, it might accidentally delete your files, leak your private data, or get tricked by a hacker. Scientists are trying to build a "food critic" that can taste the dish and compare it to the menu to catch these lies before the agent serves them up. This is the challenge of checking if what a skill says it does matches what it actually does.

Enter SkillConsist, a new digital detective designed to solve this exact mystery. The researchers behind this tool realized that previous methods were like trying to compare a one-sentence menu description to a 50-page recipe book without a clear way to match them up. The menu might say "Make Tacos," while the recipe involves chopping onions, grilling meat, warming tortillas, and mixing salsa. A simple word-for-word check would fail because the "Make Tacos" promise is one big idea, but the kitchen work is a chain of many small steps.

SkillConsist tackles this by acting like a super-organized librarian who builds two separate maps: one for the menu promises and one for the kitchen actions. First, it uses a smart AI to sort through the messy mix of text and code, separating the "what we promise" from the "what we actually do." Then, it draws these as graphs—think of them as flowcharts where every step is a node and every connection is a line.

The magic happens in the next step: Bidirectional Graph Alignment. Imagine trying to match a single, big "Make Tacos" bubble on the menu map to a whole cluster of connected bubbles on the kitchen map. SkillConsist doesn't just look for a single match; it expands outward, following the lines in the kitchen map to see if a whole group of steps (chopping, grilling, mixing) can be bundled together to perfectly explain the single menu promise. It does this in both directions, checking if every kitchen step has a menu promise and if every menu promise has a kitchen plan.

Once the maps are aligned, the detective looks for inconsistencies. It flags three types of trouble:

Conflicts: The menu says "Spicy," but the kitchen code adds "Sweet."
Unimplemented: The menu promises "Salsa," but the kitchen has no salsa ingredients at all.
Undeclared: The kitchen is secretly adding "Poison" to the mix, but the menu never mentioned it.

The researchers tested SkillConsist on a massive benchmark of 633 real-world Agent Skills, including some that were known to be tricky or malicious. The results were impressive: the tool correctly identified inconsistent skills 87.93% of the time (a score called F1), which was a huge jump of over 20 percentage points compared to the best previous methods. It also managed to pinpoint exactly where the lie was hiding in the code about 62% of the time.

Perhaps most importantly, the paper shows that this isn't just about fixing typos; it's about safety. When the researchers used SkillConsist to screen for malicious skills, it helped catch more dangerous tools without needing to run them in a risky environment first. By catching the mismatch between the promise and the reality, SkillConsist gives us a way to trust our digital assistants a little bit more, ensuring that when the computer says it's making tacos, it's not actually serving up something dangerous.

Technical Summary: SkillConsist

Problem Statement

Agent Skills provide reusable workflows for Large Language Model (LLM) agents, defined by a declaration (what the Skill claims to do) and an implementation (what it actually executes). Inconsistencies between these two sides—such as declared but unimplemented behaviors, implemented but undeclared behaviors, or conflicting behaviors—can lead to incorrect Skill selection or the invocation of dangerous, out-of-scope actions.

Existing detection methods face two primary challenges in this domain:

Mixed Content: Declaration and implementation behaviors are often intermingled within natural language instructions and code fragments, making it difficult to distinguish between a claim and an execution step.
Granularity Mismatch: A concise declaration may correspond to multiple connected implementation steps, while a single implemented behavior might be described across multiple scattered declarations.

Current approaches, which often rely on comparing predefined security taxonomies or using LLMs to match metadata with code, struggle to handle these mixed roles and varying granularities effectively.

Methodology: SkillConsist

SkillConsist addresses these challenges through a three-stage pipeline: Role Separation and Behavior Record Extraction, Behavior Graph Construction, and Bidirectional Graph Alignment and Differencing.

1. Role Separation and Behavior Record Extraction

The system first disentangles mixed content to identify semantic roles.

LLM-based Separation: An LLM (DeepSeek-V4 Pro) processes package fragments to separate declaration and implementation facts into structured behavior records ( $K = \langle C, O, R, E \rangle$ ), where $C$ is the condition, $O$ is the affected object, $R$ is the effect, and $E$ is the source location.
Static Analysis: Program analysis supplements these records by extracting implementation details from source code, configurations, and controlled execution (e.g., public entries, inputs, outputs, and side effects).
Output: Two merged, source-linked sets: Declaration records $D(S)$ and Implementation records $M(S)$ .

2. Behavior Graph Construction

To handle granularity differences, records are organized into two separate directed graphs: the Declaration Graph ( $G_D$ ) and the Implementation Graph ( $G_M$ ).

Nodes: Include record nodes, object nodes, public-entry nodes, and evidence nodes.
Edges: Six typed relations connect nodes, including entry-to-record, record-to-object, evidence-to-record, and inter-record dependencies (result-to-input, result-to-condition, condition-to-condition).
Behavior Groups: Connected subgraphs within each graph that jointly express a single behavior are identified as behavior groups ( $B_X(S)$ ). These groups serve as the atomic units for comparison.

3. Bidirectional Graph Alignment and Differencing

This stage aligns behavior groups across the two graphs to detect inconsistencies, handling the granularity mismatch by searching in both directions ( $D \to M$ and $M \to D$ ).

Anchor and Expansion: Starting from an anchor record in a source behavior group, the system searches the target graph for candidate subgraphs. It expands these candidates along behavior relations until they potentially cover the source behavior.
Subgraph Completion: Candidates are validated against four completeness predicates: Condition, Input, Result, and Path. A candidate is aligned only if it satisfies binding constraints (compatible owners, entries, scopes) and provides all required fields and paths.
Proof and Differencing:
- Supported: Evidence confirms all conditions and effects match.
- Contradicted: Evidence shows incompatible fields (e.g., different default values).
- Not-Applicable: No obligation exists.
- Unknown: Evidence is incomplete.
Coverage Checking: If no aligned candidate is found, the system generates a coverage certificate to verify that the search space was exhaustive. This prevents false positives in "missing" behaviors.
Inconsistency Outputs:
- Conflict: A required declaration has a contradicted counterpart.
- Unimplemented: A required declaration has no counterpart.
- Undeclared: An in-scope implementation has no declaration counterpart.

Key Contributions

Bidirectional Graph Alignment Formulation: The paper formulates package-level consistency as a bidirectional graph alignment problem, allowing one behavior on either side to correspond to multiple connected records on the other, thereby mitigating inherent granularity differences.
SkillConsist System: A novel system that extracts behaviors from mixed content, constructs separate declaration and implementation graphs, and locates inconsistencies through relation-guided subgraph search, differencing, and coverage checking.
Benchmark Construction: The creation of a manually reviewed benchmark of 633 Agent Skills (from ClawHub and Skill-Inject) containing 319 inconsistent and 314 consistent packages, with 442 localized inconsistency annotations.

Experimental Results

The system was evaluated on the 633-Skill benchmark against four baselines (LLM, LLM+RAG, Adapted C4RLLaMA, Adapted SKILLSCOPE).

Package-Level Detection: SkillConsist achieved 86.85% precision, 89.03% recall, and 87.93% F1. This represents a 20.43 percentage point improvement in F1 over the best baseline.
Inconsistency Localization: SkillConsist achieved 67.60% precision, 58.14% recall, and 62.52% F1.
Ablation Studies: Removing role separation or subgraph expansion significantly degraded performance, confirming their necessity for recovering valid counterparts. Removing coverage checking increased recall but drastically reduced precision due to false positives.
Malicious Skill Screening: When applied to 133 Skill-Inject packages, inconsistency risk signals improved the recall of malicious skill detection by 11.90% (High risk gate) and 26.19% (Medium/High risk gate) with minimal redundant dynamic verification.

Significance and Claims

The paper claims that SkillConsist advances the state of the art by moving beyond simple text-code comparison to a structural, graph-based approach that explicitly handles the mixed nature of Agent Skill content and the granularity gap between declarations and implementations.

By separating roles and using bidirectional alignment, SkillConsist provides source-located findings rather than just package-level flags. The authors assert that this capability is crucial for security, as it allows for the detection of undisclosed dangerous behaviors (e.g., a Skill claiming to sync email but actually forwarding files to an external endpoint) and improves the reliability of agent workflows. The results suggest that bidirectional graph alignment is a robust method for establishing correspondences in complex, mixed-media software artifacts, outperforming retrieval-augmented generation and static security property matching in both detection and localization tasks.