🤖 AI

Counsel: A Meta-Evaluation Dataset for Agentic Tasks

This paper introduces Counsel, the first public dataset of human-verified meta-evaluations for LLM-as-a-judge critiques on agentic tasks, which enables the calibration and improvement of automated evaluators by analyzing their alignment with human judgments on error location and reasoning quality.

Original authors: Sashank Pisupati, Henry Broomfield, Eujeong Choi, Antonia Calvi, Charlie Wang, Roman Engeler, Max Bartolo, Patrick Lewis

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sashank Pisupati, Henry Broomfield, Eujeong Choi, Antonia Calvi, Charlie Wang, Roman Engeler, Max Bartolo, Patrick Lewis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a team of AI assistants (agents) to handle complex jobs, like managing a customer's perfume order or writing computer code. As these jobs get harder, checking if the AI is doing a good job becomes a massive headache.

The Problem: The "Human Bottleneck"
Think of an AI agent as a student taking a long, multi-step exam. To grade the exam perfectly, a human teacher used to have to read every single step the student took.

The Reality: For complex tasks, this takes hours. If you have 1,000 exams, you need a small army of teachers working for weeks.
The Shortcut: To save time, companies started using "AI Teachers" (called LLM-as-a-Judge) to grade the exams automatically. These AI Teachers write comments like, "You forgot to check the user's ID first!" or "Good job finding the file!"

The New Problem: Are the AI Teachers Reliable?
Here's the catch: We started trusting these AI Teachers without checking if they were actually good at grading.

Sometimes an AI Teacher spots a real mistake and explains it perfectly.
Sometimes it spots a mistake that isn't actually a mistake (a false alarm).
Sometimes it spots a real mistake but gives a confusing or wrong explanation for why it's a mistake.

Until now, we didn't have a way to measure how good these AI Teachers were at their specific job of writing critiques. We only knew if they got the final "Pass/Fail" right, but not if their reasoning was sound.

The Solution: "Counsel"
The authors created a new dataset called Counsel. Think of this as a "Teacher's Teacher" dataset.

The Setup: They took two real-world scenarios:
- Customer Service: An AI trying to help a customer exchange a perfume.
- Coding: An AI trying to analyze financial data files.
The Actors:
- The Student: An AI agent trying to do the task.
- The AI Teacher (Judge): An AI that watches the student and writes a critique (e.g., "Error! You skipped a step").
- The Human Expert (Meta-Evaluator): A real human who reads the AI Teacher's critique and decides:
  - "Spot on": You found the right mistake and explained it perfectly. ✅
  - "Correct location, poor reasoning": You found the right mistake, but your explanation was weak or wrong. ⚠️
  - "Should not have flagged": You thought there was a mistake, but the student was actually fine. ❌

What They Found
By having humans grade the AI Teachers, they discovered:

Smarter is Better: More powerful AI models make better Teachers.
Thinking Harder Helps: When the AI Teacher is forced to "think" longer and more deeply before grading, it makes fewer mistakes.
The Best Teacher: The strongest AI Teacher they tested agreed with human experts about 88% of the time on where the error was, and 65% of the time on the reasoning.

Why This Matters
This dataset is like a "training manual" for building better AI Teachers. Instead of just hoping an AI is good at grading, we can now use this data to:

Test how good a new AI Teacher is.
Train AI Teachers to write better, more accurate critiques.
Filter out bad critiques so developers only see the high-quality ones.

In a Nutshell
The paper introduces a way to grade the graders. Just as you wouldn't hire a teacher who can't explain why a student got a question wrong, the AI community needs a way to ensure its automated evaluators are actually giving useful feedback. Counsel provides the first public library of these "grades on grades" to help build more reliable AI systems.

Technical Summary: Counsel – A Meta-Evaluation Dataset for Agentic Tasks

Problem Statement

As agentic systems are deployed to handle increasingly complex, multi-step tasks, the evaluation of their execution trajectories has become a critical bottleneck. Human annotation of a single agent trajectory on popular benchmarks (e.g., SWE-bench) can take hours, making it difficult to scale evaluations for performance measurement or training data curation. Consequently, the field has shifted toward automated approaches, specifically LLM-as-a-Judge (LLMJ), to critique agents at both the process level (step-by-step) and outcome level.

However, a significant gap exists: while LLMJs are widely used for data curation, training, and deployment, the soundness of their critiques is rarely measured. Existing meta-evaluation efforts typically restrict themselves to comparing final judgments (scores/preferences) against human ground truth. They lack granular, process-level data to determine if an LLMJ correctly identified where an error occurred and whether its reasoning for that error was sound. This absence of meta-evaluation data hinders the ability to directly calibrate, improve, or train LLMJs to produce higher-quality diagnostic feedback.

Methodology

The authors introduce Counsel, the first public dataset of meta-evaluations for agentic tasks. The dataset construction involves a three-stage pipeline:

1. Agent Trajectory Generation

The authors generated agent trajectories using open-weight models across two distinct benchmarks:

$\tau$ -bench (Retail): A customer support benchmark simulating multi-turn tool-agent-user interactions. The authors used the retail subset (115 tasks).
DA-Code: A data science code generation benchmark involving complex workflows (cleaning, analysis, modeling) in a sandboxed environment. The authors used the Data Insights and Data Manipulation subsets (50 tasks).

Agent Models: To ensure heterogeneous failure modes, trajectories were generated using:

GPT-OSS-20B: A 20B-parameter mixture-of-experts (MoE) model with medium reasoning capabilities.
Qwen3-235B-A22B-Instruct-2507: A 235B-parameter MoE model with no explicit reasoning (System 1 thinking).

2. Judge Critique Generation

Each step (span) of the agent trajectories was evaluated by open-weight LLMJs. The judges received the same context available to the agent at that step (no future information) and were tasked with:

Identifying if an error occurred in the current step.
Providing a concise, human-readable critique explaining the error.
Categorizing the error type (e.g., reasoning, tool use, workflow).

Judge Models: To induce systematic variation, the authors employed:

GPT-OSS-120B: Evaluated in both low and high reasoning settings.
Qwen3-235B-A22B-Instruct-2507: Used as a non-reasoning baseline.
The study included same-family (e.g., GPT judging GPT) and cross-family pairings to analyze self-preference bias.

3. Human Meta-Evaluation

Human annotators (three skilled data science professionals with 10+ years of experience) reviewed the judge outputs. Crucially, annotators had access to the full trajectory (past and future steps), unlike the judges who only saw the current context.

Annotation Schema: Annotators labeled each flagged span into one of three categories:

Spot on: The error location is correct, and the reasoning is correct.
Correct location but poor reasoning: The error location is identified correctly, but the explanation is flawed or inadequate.
Should not have flagged: The judge incorrectly identified an error (both location and reasoning are wrong).

Annotators achieved a Krippendorff's $\alpha$ of 0.78, indicating high inter-annotator agreement.

Key Contributions

Counsel Dataset: A public, permissively licensed dataset containing 1,131 human meta-annotations of judge critiques across 225 unique agent trajectories.
Granular Meta-Evaluation: Unlike prior work focusing on final scores, Counsel provides labels for both error localization and reasoning quality, enabling the evaluation of the diagnostic utility of LLMJs.
Open-Weight Ecosystem: The entire pipeline (agents, judges, and dataset generation) relies on open-weight models, facilitating broad community use and reproducibility.

Results and Analysis

The authors present several key findings from the dataset analysis:

Judge Performance Variance:
- Reasoning Effort: Judges with higher reasoning effort (GPT-OSS-120B:high) achieved significantly better human agreement than their low-reasoning counterparts. The strongest judge achieved ~88% agreement on error location and ~65% agreement on reasoning quality.
- Domain Specificity: Critique precision varied by domain. For instance, Qwen3 achieved 43% "Spot on" precision on DA-Code but 63% on $\tau$ -bench retail, suggesting that judge effectiveness is environment-dependent.
Self-Preference Bias: The study analyzed whether judges were more lenient toward agents of the same model family. Statistical tests (one-sided normal tests for proportions) found no significant self-preference bias in any of the judges evaluated.
Error Localization vs. Recall: The dataset design prioritizes precision (reviewing only flagged errors) over recall. While the dataset allows for relative recall comparisons (by comparing true positives across judges), it does not measure the rate at which judges fail to flag existing errors.

Significance and Claims

The paper positions Counsel as a foundational resource for the rigorous study and improvement of LLM-based evaluators. The authors claim the dataset enables three primary use cases:

Evaluating Critique Quality: Moving beyond final score agreement to assess whether judges correctly localize errors and provide high-quality reasoning.
Training Meta-Judges: Optimizing models that score, rank, or filter judge outputs to select higher-quality critiques.
Training Judges: Using meta-annotations as supervision or reward signals to directly optimize judges for better critique generation.

The authors emphasize that while Counsel supports the calibration and training of automated evaluators, it is intended to complement, not replace, human judgment. They note limitations, including the focus on precision over recall and the restriction to two specific agentic domains (retail and coding), which may limit generalization to other task types.

A preliminary experiment in the paper demonstrates that using Counsel's meta-annotations as few-shot in-context examples can improve agent performance in a loop, specifically when "Spot on" feedback is provided to a guardrail judge, though effect sizes were generally small.