Technical Summary: SKILLAUDIT

Problem Formulation

The paper addresses the challenge of skill evolution for frozen Large Language Model (LLM) agents in professional workflows. While "agent skills" (structured procedural instruction packages) effectively guide agents, they often degrade after deployment due to changing tools, APIs, and edge cases.

Existing methods for refining these skills rely on privileged ground-truth feedback, such as:

• Held-out validation scores or hidden test outcomes.
• Oracle pass/fail signals.
• Environment rewards or expert reference responses.

However, in realistic deployment settings, practitioners often possess only a task description ($T$), workspace data ($W$), and an initial skill ($S_0$). They lack access to external scoring functions, reference solutions, or test scripts. The central problem is: How can agent skills be improved without access to external ground-truth feedback during optimization?

The authors formalize this as a constrained optimization problem where the objective is to maximize the latent task reward $R(\tau)$ subject to a ground-truth-free constraint ($C_{gtf}$), which prohibits access to $R$ or its proxies at any point during the evolution loop.

Methodology: SKILLAUDIT

The authors propose SKILLAUDIT, a framework that evolves skills using only the task description, workspace data, and execution traces. The core mechanism is Paired Trajectory Auditing, which isolates the specific impact of a skill by comparing two executions of the same task:

1. With-skill ($\tau_w$): The agent executes the task with the candidate skill injected.
2. Without-skill ($\tau_{wo}$): The agent executes the same task without the skill.

The system derives optimization signals from the behavioral divergence between these two trajectories. The architecture consists of four key components:

1. Paired Trajectory Auditing & PACE

Raw trajectory differences are insufficient for editing. To convert them into actionable guidance, SKILLAUDIT uses Process-Aligned Contrastive Evaluation (PACE). PACE is a cluster of 12 evaluators operating across four dimensions:

• Process Adherence: Did the agent follow the skill's prescribed steps?
• Artifact Evidence: Do output files meet the constraints declared in the skill?
• Consistency: Does the skill's workflow match the task's data schema?
• Effectiveness Delta: Does the with-skill trajectory outperform the without-skill trajectory at divergence points?

Crucially, PACE outputs segment-anchored diagnostic signals. Instead of a global "pass/fail," it identifies specific passages in the skill document that caused a behavioral change (e.g., "With this paragraph, the agent took a wrong action at step X").

2. Anchor Verifier

To prevent evaluator drift and ensure structural integrity, a structural verifier is compiled once from the task specification ($T$) and fixed for the entire evolution. It encodes hard constraints derivable solely from $T$ and $W$ (e.g., file existence, format compliance, re-computable values).

• Role: It acts as a deterministic gate. If an update causes the Anchor Verifier to fail, the update is unconditionally rolled back, regardless of PACE's soft signals.

3. Dual-Strategy Evolution

Based on an initial compatibility assessment between $S_0$ and $T$, the system routes the task to one of two pipelines:

• Refine Pipeline: For skills that are broadly effective but contain noise, redundancy, or minor gaps. The strategy is subtraction-first: removing distracting content while protecting "helped" segments.
• Repair Pipeline: For skills where the core workflow conflicts with the task or actively misleads the agent. The strategy is diagnosis-driven replacement: locating harmful passages and swapping them with correct guidance (potentially derived from the without-skill trajectory).

4. Decision Logic

At each iteration, the system aggregates PACE signals and Anchor Verifier checks to produce a three-way verdict:

• Helped: Commit the update.
• Hurt: Immediately rollback (vetoed if any evaluator reports a "hurt" signal).
• Inert: Defer to the next iteration.

The loop terminates when the skill reaches a stable state (no "hurt" verdicts in recent iterations, verifier passes, and no actionable targets remain).

Key Contributions

1. Ground-Truth-Free Skill Evolution: The formulation of skill evolution under the realistic constraint $C_{gtf}$, where no hidden tests, reference solutions, or oracle signals are available.
2. Paired Trajectory Auditing: A mechanism to derive optimization signals by isolating the skill's effect on agent behavior through controlled contrastive execution.
3. Process-Aligned Diagnosis: The development of PACE, which maps trajectory divergences to specific skill passages, combined with a fixed Anchor Verifier to ensure structural stability.
4. Dual-Strategy Routing: A mechanism that distinguishes between skills needing noise reduction (Refine) and those needing core replacement (Repair).

Experimental Results

The framework was evaluated on 89 containerized tasks across 8 professional domains (including Software Engineering, Natural Science, and Finance) using the SkillsBench benchmark.

• Performance: SKILLAUDIT achieved an average task reward of 73.9%.
• Comparisons:
  • Outperformed the agent with no skills (40.9%) by +33.0 percentage points.
  • Outperformed the static expert skill included in the benchmark (56.7%) by +17.2 percentage points.
• Domain Variance: The system showed the largest gains in Software Engineering (+38.5 pp) and Office & White Collar (+26.7 pp). In Finance & Economics, it matched the static baseline but did not exceed it.
• Recovery: The system successfully recovered 43% of tasks where the initial skill failed (reward < 0.5) and perfectly protected 92% of tasks where the initial skill was already working.

Significance and Limitations

The paper claims that SKILLAUDIT demonstrates that agent skills can be autonomously improved without external ground truth, provided the task correctness leaves an observable structural trace.

• Observability Boundary: The authors identify that success is governed by observability, not domain. Skills encoding "executable, observable knowledge" (e.g., API usage, mathematical formulas) evolve well because errors produce visible traces (stack traces, wrong numbers). Conversely, skills encoding "domain procedure" (where removing a step leaves no structural trace) are difficult to evolve without ground truth.
• Nature of Edits: The analysis reveals that successful evolution tends to transform skills from "tutorials" into "verifier-observable execution contracts." It often involves pruning off-domain content, de-hardcoding paths, and adding missing I/O contracts.
• Modest Claims: The authors acknowledge that the system cannot synthesize missing domain knowledge if the auditor cannot verify the result (e.g., it cannot fix a financial model if it cannot verify the correctness of the number, only the existence of the file). The system recovers failures by making the skill's contract observable, not by inventing new domain knowledge.

In summary, SKILLAUDIT provides a practical pathway for maintaining and evolving agent skills in real-world deployments where external validation infrastructure is unavailable, relying instead on the internal consistency and structural constraints of the task itself.