🤖 AI

SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution

This paper introduces SkillTV-Bench, a comprehensive benchmark for evaluating skill-aware trajectory verification in LLM agents, alongside SkillTV-Evolve, a method that refines reusable judge skills to significantly improve both verification accuracy and trajectory selection success rates.

Original authors: Zhi Han, Chenxi Zeng, Liuhaichen Yang, Zihan Guo, Ming Zhou, Yang Li

Published 2026-08-07

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhi Han, Chenxi Zeng, Liuhaichen Yang, Zihan Guo, Ming Zhou, Yang Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers don't just chat with you but actually do things. They can open files, write code, browse the web, and even control robots to solve complex, multi-step problems. These are called "AI Agents." But here's the tricky part: when an AI agent tries to fix a broken website or plan a trip, it doesn't just give you one final answer. It takes a long journey, making hundreds of tiny moves, calling tools, and checking its work along the way.

The big question for scientists is: How do we know if the AI actually did a good job? In the past, we just looked at the final result. If the website looked fixed, we said "Great job!" But what if the AI got there by taking shortcuts, or if it fixed the wrong thing and broke something else in the process? We need a way to watch the whole journey, not just the finish line. This is where "judges" come in—special AI systems trained to review the agent's work. But these judges are currently struggling. They often get fooled by shiny final answers and miss the hidden mistakes that happened along the way, especially when the agent is using special "skills" (like a toolbox of pre-programmed tricks) to get the job done.

This paper introduces a new way to test and train these judges. The researchers created a giant playground called SkillTV-Bench, which is like a massive TV show marathon featuring 681 different episodes of AI agents trying to solve real-world tasks across 11 different fields, from cybersecurity to cooking. The twist? The judges aren't just watching a video; they are given the agent's "skills manual" and access to the actual files and logs the agent created. This lets them see exactly what the agent was supposed to do and check if it followed the rules.

The team found that even the smartest current judges are getting tripped up. They often say "Pass" to agents that look good on the surface but actually failed the mission. To fix this, the authors built a system called SkillTV-Evolve. Think of this as a "coach" for the judge. Instead of just guessing, the judge is given a dynamic checklist (called a "JudgeSkill") that tells it exactly what to look for, where to look, and what evidence proves the agent succeeded or failed. If the judge makes a mistake, the system automatically rewrites the checklist to prevent that error next time.

The results were impressive. By using this evolving checklist, the judge's accuracy jumped by nearly 15 percentage points. But the real magic happened when they used this better judge to pick the best attempts from a group of AI agents. Imagine an AI trying to solve a puzzle 10 times and producing 10 different attempts. A bad judge might pick a "fake" success, but this new, skill-aware judge could spot the one real success. With this better judge, the team was able to select a successful solution 45.5% of the time when looking at 10 attempts, compared to just 22.9% with a single attempt or a weaker judge.

In short, the paper suggests that to trust AI agents, we need judges that don't just look at the final photo but inspect the whole album, using the agent's own instruction manual to spot the fakes. By giving these judges a better "rulebook" that learns from its own mistakes, we can make AI much more reliable at doing complex, real-world tasks.

Technical Summary: SkillTV-Bench and SkillTV-Evolve

Problem Statement

Large Language Model (LLM) agents have evolved from single-turn generators into interactive systems capable of long-horizon tasks involving tool use, file manipulation, and environment interaction. This shift necessitates a change in evaluation paradigms: moving from scoring final responses to verifying complete execution trajectories. However, existing judge benchmarks often rely on static trajectories or final responses, failing to account for skill-augmented agents.

For agents enhanced with task-time skills (procedural knowledge encoding tool choices, workflows, and constraints), verification is uniquely challenging. Skills dictate what evidence to inspect and which failures are task-critical. Without access to these skills and the ability to actively inspect execution artifacts (e.g., logs, files, database states), judges frequently misinterpret agent behavior, over-trust polished outputs, or miss hidden process failures. Current benchmarks rarely combine task-time skills with directly inspectable environments and multi-domain coverage required to systematically evaluate skill-aware trajectory verification.

Methodology

1. SkillTV-Bench: A Skill-Aware Verification Benchmark

To address the gap in existing evaluation, the authors introduce SkillTV-Bench, a benchmark designed for skill-aware trajectory verification.

Composition: The benchmark comprises 681 execution cases derived from 50 source tasks across 11 domains (including cybersecurity, finance, healthcare, robotics, and software engineering).
Data Structure: Each case is a self-contained tuple $(I, \tau, S, A)$ $(I, τ, S, A)$ :
- $I$ : Original task instruction.
- $\tau$ : Normalized, step-wise execution trajectory.
- $S$ : The set of task-time skills available to the agent.
- $A$ : Inspectable execution artifacts (files, logs, environment states).
Construction: Data is filtered and normalized from SkillsBench. Cases are partitioned into a development set (478 cases) for skill evolution and a disjoint evaluation set (203 cases) for final testing.
Differentiation: Unlike prior benchmarks that focus on response pairs or static text, SkillTV-Bench supports agentic interaction, allowing judges to actively inspect evidence within an external environment.

2. SkillTV-Evolve: Externalized Verification Knowledge

The authors propose SkillTV-Evolve, a framework to improve judge performance without updating underlying model parameters.

JudgeSkill Concept: Verification knowledge is externalized as a reusable, human-readable JudgeSkill. This skill guides an agent judge to:
1. Plan: Generate an inspection plan specifying what to check and where to find evidence.
2. Execute: Perform targeted inspections on artifacts and skills.
3. Verdict: Issue an evidence-grounded judgment with a rationale.
Automated Evolution Loop:
- The system analyzes misjudged cases (false positives/negatives) from a development pool.
- It synthesizes "rewrite briefs" that compress failure analysis into targeted revision goals (e.g., false-pass hardening).
- An optimizer proposes revisions to the JudgeSkill text.
- Gating: Candidate revisions are evaluated on a fixed 36-case development gate. A candidate is promoted only if it improves a composite score ( $G$ ) balancing balanced accuracy, recall for failures, and precision for passes.
Outcome: This process iteratively refines the JudgeSkill, turning judgment errors into reusable verification procedures.

Key Contributions

SkillTV-Bench: A 681-case benchmark spanning 11 domains that supports multi-domain, agentic, evidence-grounded, and skill-aware evaluation. It expands domain coverage, agentic interaction, and skill awareness beyond existing benchmarks by combining task-time skills with inspectable artifacts for judge evaluation.
SkillTV-Evolve: A plan-driven framework that externalizes verification knowledge into a JudgeSkill. It features an automated evolution loop that improves verification accuracy through iterative refinement of the skill text, without model retraining.
Empirical Validation: Experiments demonstrating that connecting benchmark-grounded evaluation with JudgeSkill evolution significantly improves both trajectory verification and downstream rollout selection.

Experimental Results

RQ1: Benchmark Insights

SkillTV-Bench reveals systematic weaknesses in existing judges:

Performance Ceiling: Frontier models (GPT-5.2, Claude Sonnet 4.6, Gemini 3.1) achieve balanced accuracies between 0.478 and 0.586 on the benchmark.
Error Patterns: The dominant failure mode is false acceptance. Judges frequently accept plausible-but-failed trajectories (e.g., Claude Sonnet 4.6 has high recall but low precision).
Insufficiency of Tools: Merely providing agents with tools and environment access (without a dedicated JudgeSkill) yields a balanced accuracy of only 0.568, indicating that environment interaction alone does not solve the verification problem.

RQ2: Benchmark-Grounded Improvement

SkillTV-Evolve successfully converts benchmark errors into verification knowledge:

Accuracy Gain: The refined JudgeSkill increases the same agent judge's balanced accuracy from 0.568 to 0.634 and overall accuracy by 14.8 percentage points (from 43.8% to 58.6%).
Domain Coverage: Improvements are observed across 7 of 9 evaluation domains, with significant gains in media/content (+36.7 pp), cybersecurity (+26.7 pp), and manufacturing (+23.1 pp).
Iterative Progress: Accuracy on the full evaluation set improved monotonically from version v1 (50.8%) to v4 (58.6%).

RQ3: Downstream Rollout Selection

Improved verification translates to better selection of successful trajectories in offline rollout pools:

Success Rate: With a budget of 10 rollouts, the refined JudgeSkill (v4) achieves a selected-trajectory success rate of 45.5%.
Comparison: This represents a 22.6 percentage point gain over the single-rollout baseline (22.9%) and outperforms the best non-skill baseline (Rubric-based judge at 38.4%) by 7.1 points.
Mechanism: The improvement is driven by a reduction in false acceptances. By filtering out plausible failures more effectively, the verifier preserves a "cleaner" pool of candidates, increasing the probability of selecting a genuinely successful trajectory.

Significance and Claims

The paper posits that trajectory verification should be viewed not merely as passive evaluation, but as the mechanism that converts diverse agent rollouts into reliable task outcomes.

Measurability: SkillTV-Bench makes the problem of skill-aware verification measurable by requiring judges to reconcile instructions, processes, skills, and artifacts.
Externalizability: SkillTV-Evolve demonstrates that the complex knowledge required for verification can be externalized as an explicit, auditable procedure (JudgeSkill) and refined from errors without updating model parameters.
Practical Consequence: The study validates that evolving an explicit JudgeSkill yields consistent gains in verification capability, which directly translates to higher success rates in selecting agent trajectories.

The authors note a limitation: the current loop improves the judge but does not yet feed evidence-grounded feedback back to the task agent to improve future executions. They suggest that closing this gap would create a data flywheel where verified trajectories expand the benchmark and drive further agent and judge evolution.