🤖 AI

Cross-Layer Misalignment Detection in Agent Skills: A Progressive Loading-Aware Contrastive Learning Approach

This paper introduces Progressive Loading-Aware Hierarchical Contrastive Learning (PL-HCL), an LLM-based framework that effectively detects cross-layer misalignment between the descriptions and actual behaviors of Agent Skills, achieving a significant performance boost in Macro-F1 scores from 0.45 to 0.87–0.89 on a large-scale dataset of open-source skills.

Original authors: Chengjun Zhang, Yang Gao, Jianna Hur, Jingjing Zhang, Sagar Samtani

Published 2026-07-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chengjun Zhang, Yang Gao, Jianna Hur, Jingjing Zhang, Sagar Samtani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're browsing a digital marketplace for "Agent Skills." These are like little, reusable robot helpers that you can download to make your AI smarter. You see a skill named "Super Safe Productivity Assistant." The description sounds perfect, right? But what if, hidden deep inside the code, that same skill is actually a sneaky spy trying to steal your passwords or a chaotic robot that ignores all your rules?

This is the problem of Cross-Layer Misalignment. It's like buying a toy that says "Builds Castles" on the box, but when you open it, the instructions tell you to "Demolish the house," and the bricks are actually made of lava.

The Detective's New Superpower

The researchers at Indiana University Bloomington realized that current AI models are terrible at spotting these "box vs. contents" mismatches before you actually run the skill. They tried to see if standard AI models (the "base" models) could just read the description and guess if it was a liar. The result? Not even close. Even the smartest cybersecurity-trained models mostly guessed "It's safe" just because most skills are safe, failing to catch the liars. They got about 45% accuracy on the tricky stuff, which is basically a coin flip.

So, the team built a new training method called PL-HCL (Progressive Loading-Aware Hierarchical Contrastive Learning). Think of this as a "Truth-Training Camp" for AI.

How the Training Camp Works

Instead of just reading the whole package at once, the AI learns in two stages, mimicking how a human would inspect a skill:

Stage 1: The "Teaser" View. The AI first looks at the "box" (the metadata and description) and the "instruction manual" (the procedural steps). It learns the language and format of these skills.
Stage 2: The "Full Reveal" View. Then, the AI gets to see the "actual bricks" (the code, scripts, and resources).

The magic happens in the training itself. The researchers didn't just show the AI real skills. They created a game of "Spot the Fake."

The Real Deal: They showed the AI a skill where the description matched the code.
The Swap: They took the description of a "Safe Assistant" and glued it onto the code of a "Virus."
The Lie: They took a "Safe Assistant" description and tweaked it to sound slightly exaggerated or wrong, while keeping the code the same.

The AI had to learn that the "Real Deal" is a match, but the "Swap" and the "Lie" are mismatches. It learned to compare the claim on the surface with the evidence in the deep layers.

The Results: From Coin Flip to Detective

When they tested this new method on a "Challenge Set" of over 1,400 real-world skills (including 294 that were actually misaligned), the results were a massive jump.

Before Training: The best base models could only identify about 14% of the misaligned skills correctly (a metric called F1). They were mostly ignoring the liars.
After PL-HCL Training: The model's ability to catch the liars skyrocketed to 78% to 81% (depending on the specific AI model used). The overall "score" for balancing safety and accuracy jumped from around 0.52 to 0.87–0.89.

The paper shows that simply teaching the AI to understand the format of these skills (Stage 1) wasn't enough; it still couldn't spot the lies. It needed the specific "Spot the Fake" training (Stage 2) to truly understand the mismatch.

What This Means (and What It Doesn't)

The authors suggest this is a powerful new tool for pre-execution screening. Imagine a plugin for your computer that scans a skill before you download it, checking if the "box" matches the "contents." If they don't match, it warns you: "Hey, this description says 'TypeScript helper,' but the code is actually trying to download a recipe from a random website!"

However, the paper is very clear about what this does not do:

It doesn't run the code. It can't see if a skill breaks your computer while it's running. It only looks at the files and text available before you hit "execute."
It doesn't solve every security problem. It specifically targets the mismatch between what is promised and what is delivered.
The results are based on a specific dataset of open-source skills from a platform called SkillsMP and GitHub. The authors note that we don't yet know if this works exactly the same way for closed, private, or enterprise skills.

In short, the paper suggests that by teaching AI to play a rigorous game of "matching claims to evidence," we can build a much better filter for the digital tools of the future, catching the liars before they ever get a chance to run.

Technical Summary: Cross-Layer Misalignment Detection in Agent Skills

Problem Definition

As Large Language Model (LLM) agents evolve into agentic AI systems, they increasingly rely on Agent Skills—reusable artifacts containing natural-language metadata, procedural instructions, and execution-time resources. A critical vulnerability arises from the hierarchical nature of these skills: users and agents often select skills based on surface-level metadata (descriptions, tags, names) before the underlying instructions and resources are fully loaded or executed.

The paper identifies Cross-Layer Misalignment as the discrepancy between a skill's user-facing claims (metadata) and its true behavior (instructions and resources). This misalignment manifests in two forms:

Malicious Misalignment: Metadata masks harmful behaviors (e.g., prompt injection, credential leakage, insecure command execution) present in deeper layers.
Benign Misalignment: Metadata exaggerates or inaccurately describes capabilities that the underlying package does not actually support.

Current evaluation paradigms focus on runtime performance (task success, tool proficiency) or specific security vulnerabilities within isolated artifacts. They fail to assess the consistency between layers prior to execution, leaving a gap where trust decisions are made on partial information.

Methodology: PL-HCL Framework

To address this, the authors propose Progressive Loading-Aware Hierarchical Contrastive Learning (PL-HCL). The framework treats Agent Skills as layered artifacts ( $S = (M, I, R)$ ) and trains models to verify if the metadata ( $M$ ) is substantiated by the instructions ( $I$ ) and resources ( $R$ ).

1. Dataset Construction

Corpus: A normalized corpus of 264,937 open-source Agent Skills from SkillsMP, linked to GitHub repositories. Each skill is structured into three layers: Metadata, Instructions, and Resources.
Challenge Set: A human-verified, held-out evaluation set of 1,444 skills (1,150 aligned, 294 misaligned). Crucially, the misaligned examples are naturally occurring "in-the-wild" cases, not synthetically generated, to test generalization.

2. Two-Stage Continued Pretraining (CPT)

Before contrastive learning, the base LLM undergoes a curriculum-inspired adaptation to the skill format:

Stage 1 (Short View): Trains on Metadata + Instructions ( $M, I$ ), simulating the initial inspection phase.
Stage 2 (Full View): Trains on Metadata + Instructions + Resources ( $M, I, R$ ), exposing the model to the full package context.
Goal: To adapt the model to the vocabulary, formatting, and hierarchical structure of Agent Skills, rather than immediately forcing it to judge consistency.

3. Progressive Loading-Aware Hierarchical Contrastive Learning (PL-HCL)

This component explicitly learns cross-layer consistency by contrasting views:

Positive View ( $P_i$ ): Aligned layers from the same skill: $(M_i, I_i, R_i)$ .
Negative Views:
- Swap Negative ( $N^A_i$ ): Metadata from a different skill paired with the original instructions/resources: $(M_j, I_i, R_i)$ where $j \neq i$ .
- Corruption Negative ( $N^B_i$ ): Perturbed/misleading metadata paired with original layers: $(\tilde{M}_i, I_i, R_i)$ .
Objective: The model learns to score the aligned view higher than the mismatched variants, effectively learning a "claim-versus-evidence" signal. This training occurs in two sub-stages corresponding to the short and full views used in CPT.

Key Contributions

Problem Formalization: Defined cross-layer misalignment as a pre-execution, artifact-level evaluation problem, distinguishing it from standard runtime security or task-performance evaluation.
Dataset: Constructed a large-scale open-source Agent Skill corpus and a human-verified challenge set specifically for aligned/misaligned evaluation.
Framework: Introduced PL-HCL, a contrastive pretraining framework that learns cross-layer consistency.
Empirical Findings: Demonstrated that while skill-format adaptation (CPT) is necessary, it is insufficient for detection; explicit contrastive learning is required to identify misalignment.

Experimental Results

The framework was evaluated on the Challenge Set using general (e.g., Llama-3.1-8B) and cybersecurity-tuned (e.g., Foundation-Sec-8B) backbones across zero-shot, two-shot, and five-shot settings.

Base LLMs (E0): Unadapted models performed poorly on the misaligned class, achieving Macro-F1 scores around 0.45–0.52 and Misaligned F1 ( $F1_m$ ) below 0.15. They exhibited a strong bias toward classifying plausible packages as "aligned."
Continued Pretraining Only (E1): Two-stage CPT improved adaptation to the skill format but yielded limited gains in misalignment detection. Macro-F1 remained near 0.48–0.52, and $F1_m$ stayed below 0.15. This indicates that familiarity with the format does not equate to the ability to detect inconsistencies.
PL-HCL (E2): The full framework significantly improved performance.
- On Foundation-Sec-8B, Macro-F1 rose from ~0.52 (CPT-only) to 0.889, and $F1_m$ jumped from 0.150 to 0.817 (two-shot setting).
- On Llama-3.1-8B, Macro-F1 reached 0.872 and $F1_m$ reached 0.788.
Generalization: Despite being trained on synthetic negative pairs (swapped/corrupted metadata), PL-HCL successfully generalized to naturally occurring, "in-the-wild" misaligned skills in the Challenge Set.

Significance and Claims

The paper claims that PL-HCL offers a scalable, pre-execution screening tool for the agentic AI ecosystem. Its significance lies in three areas:

Shift in Evaluation Paradigm: It moves the focus from "does the tool work?" (runtime) to "is the tool honest about what it is?" (pre-execution consistency).
Stakeholder Utility:
- Marketplaces: Can use PL-HCL as a governance mechanism to flag high-risk skills for manual review or assign verification badges before listing.
- Users: Enables lightweight client-side screening to warn users when a skill's description contradicts its code or resources.
- Analysts: Provides a method to triage large collections of skills by identifying clusters of metadata-behavior inconsistencies.
Design Principle: The authors argue that for any progressively loaded, hierarchical digital artifact (e.g., browser extensions, MCP servers), trust decisions made prior to full execution require models to explicitly learn the consistency between surface claims and deeper evidence, rather than relying solely on generic language modeling or keyword-based scanning.

The authors remain modest regarding limitations, noting that the approach relies on static package artifacts and cannot detect dynamic runtime behaviors, side effects, or mutations that occur only during live execution. They suggest PL-HCL should be viewed as a complementary layer to runtime sandboxing and trajectory-level evaluation.