💻 computer science

Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation

This paper presents a systematic validity audit of four major tool-calling benchmarks, revealing significant evaluator-human misalignment and reproducibility issues that undermine current leaderboard scores, and proposes a unified taxonomy of failures alongside new tools like Tool-Veritas and Harness Lab to establish more rigorous, auditable, and human-aligned evaluation standards.

Original authors: Vishvesh Bhat, Jay Vaghasiya, Muhammad Ahmed Mohsin, Asad Aali

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vishvesh Bhat, Jay Vaghasiya, Muhammad Ahmed Mohsin, Asad Aali

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a coach trying to figure out which of your athletes is the best at playing a complex video game that involves solving puzzles, managing inventory, and talking to NPCs (non-player characters). To decide the winner, you set up a series of challenges and use a scorekeeper to grade their performance.

This paper is essentially a forensic audit of the scorekeepers themselves. The authors, researchers from CoreThink AI and Stanford, asked a simple but shocking question: "Are the scores we're seeing actually measuring the athletes' skills, or are they just measuring how broken the scorekeepers are?"

Here is what they found, explained through everyday analogies.

1. The Problem: The Scorekeeper is Flaky

The researchers looked at four major "scorekeeping systems" (benchmarks) currently used to rank AI agents. They took 496 specific tasks and had human experts watch the AI play, then compared the human's judgment with the computer scorekeeper's grade.

The Result: The scorekeepers were wrong 18.5% of the time.

False Negatives: The AI actually solved the puzzle, but the scorekeeper gave it an "F" because of a tiny, irrelevant mistake (like a missing comma or a slightly different way of phrasing an answer).
False Positives: The AI failed to solve the puzzle, but the scorekeeper gave it an "A" because it didn't notice the failure.

The Analogy: Imagine a student correctly solves a math problem but writes the final answer as "42" instead of "42.". The teacher (the AI agent) knows the math is right, but the automated grading machine (the benchmark) fails the student because it demands an exact character match. Conversely, a student might get the answer wrong, but if the grading machine is confused, it might accidentally give them full credit.

2. Two Types of Broken Scorekeepers

The paper found that the scorekeepers fail in two very different ways, depending on how they are built.

Type A: The "Rigid Robot" (Deterministic Benchmarks)

These systems are like a strict security guard who only lets you in if your ID matches the photo exactly, down to the pixel.

The Flaw: They are "brittle." If the AI does the right thing but in a slightly different order, or if the database updates in a way the robot didn't expect, the robot panics and marks it as a failure.
Real-world example from the paper: An AI correctly canceled two flight reservations and calculated the cost of the remaining ones. However, the scorekeeper expected a specific number ($1,628) that included the canceled flights. Because the AI gave the correct answer for the user's request but the wrong number for the scorekeeper's rigid script, the AI was marked as failing.

Type B: The "Drunk Judge" (LLM-Judge Benchmarks)

These systems use another AI to grade the first AI. It's like asking a student to grade their own homework, but the grader is also a student who is tired and inconsistent.

The Flaw: They are "stochastic" (random). If you run the exact same test twice, the "Drunk Judge" might give two completely different scores.
Real-world example from the paper: The researchers ran the same benchmark 23 times. The score swung wildly from 57.9% to 76.8%. That's an 18.9-point difference! If you ran this test on a leaderboard, the "winner" could change just because the judge had a different mood that day, not because the AI got smarter.

3. The Consequence: A Broken Leaderboard

Because the scorekeepers are so unreliable, the current "Leaderboards" (lists ranking the best AI models) are misleading.

The Analogy: Imagine a race where the finish line moves randomly every time a runner crosses it. Sometimes the line is 10 meters back, sometimes 10 meters forward. If you tell the world who won based on one run, you are lying. The paper argues that current AI rankings often reflect the flaws of the test rather than the skill of the AI.

4. The Solution: A New Way to Grade

The authors didn't just complain; they built two new tools to fix the problem.

Tool-Veritas: The "Fact-First" Judge

This is a new scoring system that tries to get the best of both worlds.

How it works: First, it checks the facts using a rigid, unchangeable robot. Did the file get saved? Did the bank account update? If the facts are wrong, the test is over.
The Safety Net: Only if the facts are perfect does it let a human-like AI judge look at the "style" or "tone" of the answer.
The Result: This new system agreed with human experts 95.5% of the time, a massive improvement over the 69–90% agreement of the old systems.

Harness Lab: The "Replay Camera"

This is a piece of software that acts like a sports broadcast replay system.

What it does: It saves every single move the AI made, every error message, and every decision the scorekeeper made.
Why it matters: If a score looks weird, you can hit "rewind," watch the exact moment the AI or the judge messed up, and fix it. It makes the grading process transparent and auditable, rather than a "black box" where you just get a final number.

Summary

The paper concludes that we cannot trust the current "scores" of AI agents because the tests themselves are broken. To know if an AI is actually getting better, we need tests that are:

Reproducible: Running the test twice should give the same result.
Auditable: We need to be able to see why a grade was given.
Aligned with Humans: The test should agree with what a human expert thinks is a success.

The authors have released their new "Fact-First" test and their "Replay Camera" software to help the community build better, fairer benchmarks.

Technical Summary: Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation

Problem Statement

Tool-calling benchmarks have become the primary mechanism for measuring progress in language-model agents, yet their scores are frequently treated as ground truth without validating the evaluators themselves. This paper argues that current benchmark scores often reflect properties of the evaluator rather than the capabilities of the agent. The authors identify two primary failure modes across existing paradigms:

Deterministic Benchmarks: Suffer from brittle state matching, trajectory lock-in, incorrect ground-truth annotations, substring-based communication failures, and reward-basis misalignment. These often result in false negatives (valid tasks marked as failures) or false positives (invalid tasks marked as successes).
LLM-Judge Benchmarks: Exhibit rubric drift, hallucinated completions, answer-only scoring, and substantial run-to-run variance. These introduce score instability that can alter leaderboard conclusions without any change in the evaluated model.

The core problem is that existing benchmarks fail to satisfy fundamental validity requirements: correctly distinguishing successful from unsuccessful executions, faithfully capturing user intent, and producing reproducible scores under repeated evaluation.

Methodology

The authors conducted a systematic validity and reproducibility audit of four major tool-calling benchmark families: BFCL v4, τ 2-Bench, LiveMCPBench, and MCP-Atlas.

Audit Framework

The audit involved a three-stage procedure:

Reproduction: Reproducing benchmark evaluations to collect complete execution traces, tool invocation logs, evaluator outputs, and environment states.
Human Adjudication: Three independent expert annotators reviewed 496 task executions. They inspected the full interaction trajectory, tool sequences, state transitions, and final outcomes to assign a human label ( $h_i \in \{PASS, FAIL\}$ ).
Root-Cause Analysis: Comparing official benchmark labels ( $y_i$ ) against human judgments to identify disagreements ( $\delta_i = 1[y_i \neq h_i]$ ). Disagreements were categorized into false negatives and false positives and mapped to a unified failure taxonomy.

Infrastructure

The authors developed Harness Lab, an open-source system used to execute the audit. Harness Lab preserves raw artifacts, provides turn-level diagnostics, supports selective retry of failed cases, and enables repeated-run comparisons to measure score variability.

Proposed Solutions

To address identified limitations, the paper introduces:

Tool-Veritas: A configurable benchmark using a "deterministic-first" protocol. It employs deterministic state gates to verify factual outcomes (e.g., file creation, database updates) and restricts LLM judging to qualitative criteria only after deterministic gates pass. It also includes a bounded repair window to measure agent recovery.
Decomposed Metrics: A framework separating evaluation into three independent components: tool invocation correctness, task completion correctness, and outcome verification.

Key Results

Evaluator-Human Misalignment

Across 496 expert-reviewed tasks, the audit identified 92 evaluator-human disagreements, corresponding to an aggregate misalignment rate of 18.5%.

BFCL v4: 20.0% error rate (40 disagreements). Failures were dominated by instance_state_mismatch (80%), where the evaluator penalized irrelevant state differences or punctuation rather than task success.
τ 2-Bench Retail: 9.8% error rate (11 disagreements). Errors included false negatives due to brittle substring checks (e.g., rejecting a correct answer because it lacked a specific number string) and false positives where unchanged database states were incorrectly scored as success.
MCP-Atlas: 13.5% error rate (12 disagreements).
LiveMCPBench: 30.5% error rate (29 disagreements), the highest among the audited benchmarks.

Reproducibility of LLM Judges

The audit of LiveMCPBench revealed significant instability. In 23 repeated evaluations of the same 95-task setup, scores ranged from 57.9% to 76.8%, a spread of 18.9 percentage points. This variance is large enough to change leaderboard rankings, indicating that a single-run score is insufficient for fine-grained model comparison.

Tool-Veritas Performance

When evaluated on 70 tasks across six models, Tool-Veritas achieved an aggregate agreement of 95.5% with expert trace-level judgments.

All 19 disagreements were false negatives (the benchmark was stricter than the human expert).
There were zero false positives (the benchmark never passed an execution rejected by a human reviewer).
This suggests that separating deterministic verification from qualitative judging significantly reduces unsupported passes while maintaining high alignment with human intent.

Significance and Claims

The paper claims that progress in tool-using agents requires benchmarks whose evaluators are themselves reproducible, auditable, and aligned with human judgments of task success.

Validity Over Scale: The authors argue that increasing task scale or coverage is insufficient if the evaluation logic is flawed. Current scores often reflect evaluator artifacts (e.g., API instability, judge bias, rigid state matching) rather than agent capability.
Taxonomy of Failure: The work provides a unified taxonomy of evaluation failures, distinguishing between deterministic issues (brittle matching, annotation errors) and stochastic issues (rubric drift, judge variance).
Infrastructure for Debugging: By releasing Harness Lab and corrected evaluation components, the paper advocates for a shift toward transparent, inspectable benchmarking where raw artifacts and human adjudications are preserved and accessible.
Design Principle: The authors propose that future benchmarks should adopt a "deterministic-first" approach, using LLM judges only for criteria that cannot be represented as state predicates, thereby preventing fluent but incorrect responses from receiving credit.

The findings suggest that the field must move away from treating benchmark scores as absolute truth and toward a model of evaluation that explicitly accounts for and mitigates evaluator error.