Technical Summary: SkillCoach

1. Problem Statement

Large Language Model (LLM) agents increasingly rely on skills—reusable operational layers encoding Standard Operating Procedures (SOPs), domain rules, tool workflows, and validation routines. However, as skill repositories grow, agents face significant challenges in reliable skill-use within realistic environments characterized by overlapping workflows and distractor skills.

The core problem identified is that final task success (verifier pass/fail) is an insufficient signal for evaluating or training agentic skill-use. An agent may pass a verifier through:

• Trial and error rather than correct skill invocation.
• Selecting distractor skills that coincidentally yield a correct output.
• Skipping required steps or omitting final validation checks.
• Incorrectly composing workflows.

Existing evaluation methods often assume that if a skill exists, the agent can use it correctly. In reality, agents frequently fail to select the right skill, follow its prescribed steps, compose it with other skills, or reflect on the output before submission. This leads to brittle behaviors that are not reusable or scalable in enterprise settings.

2. Methodology: SkillCoach Framework

The authors propose SkillCoach, a self-evolving rubric framework designed to evaluate and enhance agentic skill-use by shifting focus from outcome-only metrics to process-level supervision.

2.1 Core Definitions

SkillCoach defines Agentic Skill-Use as a trajectory-level meta-ability comprising four distinct dimensions, kept separate from the external verifier signal:

1. Skill Selection: Evaluating whether the agent invokes appropriate gold skills while avoiding distractors.
2. Skill Following: Assessing adherence to key procedures and steps specified in the gold skills.
3. Skill Composition: Determining if multiple skills or dependent subprocesses are coordinated in a valid workflow order.
4. Skill-Grounded Reflection: Checking for explicit validation of outputs (e.g., schema, format, constraints) before final submission.

2.2 The Self-Evolving Rubric Process

SkillCoach operates in a three-stage cycle to evolve task-specific rubrics from real agent rollouts:

• Stage 1: Environment & Task Setup: Tasks are defined as skill-dependent (solvable with gold skills but difficult without). The environment includes gold skills and distractor skills (unrelated or semantically similar but inapplicable) to simulate enterprise repositories.
• Stage 2: SkillCoach Core (Evolution):
  • Initial Construction: A base rubric ($R_0$) is built from task instructions, gold skills, and oracle solutions, defining observable key steps and evidence requirements.
  • Trajectory Judging: Real agent rollouts are scored against the current rubric. Evidence is extracted (skill reads, tool calls, file edits) to score the four meta-ability dimensions.
  • Validation-Gated Evolution: An arbitration model proposes local patches (e.g., adding evidence requirements, clarifying negative cases) to the rubric. These patches undergo a hard gate (rejecting destructive changes like ignoring distractors) and a soft objective check (improving evidence coverage and consistency). Only patches that pass validation on held-out rollouts are accepted.
  • Selection: The best evolved rubric ($R_{best}$) is selected based on validation performance.
• Stage 3: Downstream Uses:
  • Trajectory Diagnosis: Pinpoints specific process-level failures (e.g., wrong skill selection, skipped steps).
  • Rubric-Filtered Training: High-quality trajectories (those passing the external verifier and scoring $\ge 0.95$ on the evolved rubric) are selected for Supervised Fine-Tuning (SFT).

3. Key Contributions

The paper claims three primary contributions:

1. Formulation of Agentic Skill-Use: Defining skill-use as a trajectory-level meta-ability (selection, following, composition, reflection) distinct from final task success.
2. SkillCoach Framework: Introducing the first self-evolving rubric framework that calibrates process rubrics from real rollouts using evidence-grounded judging and validation-gated updates, specifically designed for skill-dependent tasks with distractor-augmented libraries.
3. Dual Utility of Evolved Rubrics: Demonstrating that evolved rubrics serve both as improved evaluation instruments (exposing failures hidden by final accuracy) and as superior supervision filters for training, outperforming outcome-only SFT.

4. Experimental Results

Experiments were conducted on skill-dependent tasks with distractor-augmented libraries, comparing initial rubrics ($R_0$) against evolved rubrics ($R_{best}$).

• Rubric Quality Validation:

  • Gold-Keypoint Coverage: Increased from 71.56% to 83.70%, indicating better alignment with human-gold process evidence.
  • Usability Score: Improved from 81.53 to 94.33, suggesting clearer criteria for LLM judges.
  • Hallucination Rate: Reduced from 2.00% to 0.00%, confirming that coverage gains did not introduce unsupported constraints.
  • Filtering Consistency: Improved from 82.00% to 96.00% against human-gold trajectory judgments.

• Agentic Skill-Use Performance:

  • In "Gold + Distractors" settings, final accuracy often masked process failures. For instance, while agents achieved high final accuracy with gold skills, their Skill Selection scores dropped significantly when distractors were introduced (e.g., Qwen3.5-9B dropped from 92.0 to 44.0).
  • The four-dimensional evaluation revealed that agents often selected wrong skills or skipped steps even when the final outcome was correct.

• Training Effectiveness (SFT):

  • Rubric-Filtered SFT significantly outperformed Outcome-only SFT.
  • For Qwen3.5-9B, outcome-only SFT improved accuracy from 14.0% to 18.0%, whereas Rubric-filtered SFT (using $R_{best}$) improved it to 32.0%.
  • Ablation studies showed that key-step following was the most critical dimension for filtering, followed by composition order and reflection.

• Distractor-Boundary Analysis:

  • As the skill library scaled (up to 50k distractors), models exhibited two failure modes: degradation (performance drop) and collapse (near-total failure to select gold skills).
  • Stronger models (e.g., GPT-5.5, Opus 4.7) delayed degradation and collapse compared to smaller models, but all eventually faced reliability limits in large, overlapping libraries.
  • High-similarity distractors caused significantly more performance drops than unrelated distractors, highlighting the difficulty of distinguishing near-neighbor skills.

5. Significance and Claims

The paper posits that reliable skill-use is both measurable and trainable through skill-grounded process supervision.

• Beyond Final Accuracy: The authors argue that in enterprise settings, a successful outcome does not guarantee a reusable or robust agent. An agent that passes a verifier via trial-and-error or incorrect skill invocation introduces latency, cost, and compliance risks.
• Process Supervision: By separating process quality from outcome signals, SkillCoach provides actionable feedback for diagnosing specific failure modes (selection, following, composition, reflection) rather than just binary pass/fail results.
• Scalability: The self-evolving nature of the rubrics allows them to adapt to the complexities of growing, overlapping skill repositories without manual re-engineering, making them suitable for industrial deployment where skill libraries are dynamic.
• Training Signal: The framework demonstrates that filtering training data based on process-level rubrics yields agents with more robust skill-use behaviors than those trained solely on outcome success.

The authors conclude that while their work is limited to offline SFT and a specific set of skill-dependent tasks, the approach establishes a foundation for treating skill-use as a distinct, evaluable, and optimizable capability in agentic systems.