🔬 materials science

The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean

This paper identifies a "double measurement confound" in LLM agent benchmarks where fixed scaffolds and flawed scorers obscure true model capability, and proposes a measurement-theoretic framework with an audit-and-repair protocol to transfer execution control to models, implement ground-truth scoring, and report reliability metrics beyond the mean.

Original authors: Yonghong Zhang, Shadi Motaali, Vu Phong Dinh, Avin Piroutiniya, Jorge E. López de Vergara, Luis de Pedro, Ricardo Correia, Isabel M. Parra, Yong Xie

Published 2026-09-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yonghong Zhang, Shadi Motaali, Vu Phong Dinh, Avin Piroutiniya, Jorge E. López de Vergara, Luis de Pedro, Ricardo Correia, Isabel M. Parra, Yong Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, researchers rely on standardized tests, known as benchmarks, to measure how well large language models perform. These models are designed to act as agents, capable of using digital tools to solve complex problems, from analyzing data to booking flights. To determine which model is the best, scientists run them through these tests and compare the scores. The underlying assumption is that a higher score means a smarter, more capable machine. However, this confidence rests on a fragile premise: that the test itself is actually measuring the machine's intelligence, rather than the cleverness of the test's design or the software running it. If the test is flawed, the rankings become misleading, potentially guiding developers and businesses toward the wrong tools.

A team of researchers has uncovered a specific, hidden flaw in how these agent benchmarks are currently constructed, a problem they call a "double measurement confound." They discovered that in many popular tests, the software running the experiment is doing almost all the heavy lifting, while the artificial intelligence is merely filling in the blanks. Furthermore, the system grading the results often checks if the answer looks correct on the surface, rather than verifying if the answer is actually true. When these two issues happen together, they create a perfect illusion of performance where even a simple computer script with no intelligence can score just as highly as the most advanced artificial intelligence.

To expose this, the researchers focused on a benchmark designed to test how well models handle messy, real-world data, specifically trade statistics that contain errors, duplicates, and missing pages. In the standard version of this test, the software wrapper surrounding the model automatically handles every difficult decision. If the data source fails, the software retries the connection. If the data contains duplicates, the software removes them. If the data is spread across many pages, the software fetches them all. The artificial intelligence is only asked to decide which specific data to look for and when to stop. Because the software is doing all the critical work, the results are identical regardless of which model is used. In one striking instance, two of the world's most powerful models, along with a simple rule-based script that uses no artificial intelligence at all, all achieved nearly the same perfect score. The test was not measuring the models; it was measuring the software that ran them.

The second part of the problem lies in how the results are graded. The standard grading system in these tests looks at the format of the submission. It checks if the model provided a log of its actions and if the data structure looks neat. It does not compare the submitted data against the actual correct answer. The researchers demonstrated that a model could submit a completely fabricated set of fake records, as long as the file looked well-organized and the log was written correctly, and it would receive the same high score as a model that submitted the perfect, correct data. This means the test was rewarding the appearance of work rather than the work itself.

The researchers realized that fixing just one of these problems was not enough. If they only changed the grading system to check for truth, the models would still be tied because the software was still doing all the work. If they only removed the software's help, the flawed grading system would still hide the models' failures. They had to remove both at the same time. They stripped away the helper software, forcing the artificial intelligence to make every single decision about retrying connections, removing duplicates, and fetching pages. Then, they replaced the format-checking grader with a system that compared the model's output directly against the known correct answer.

When they applied this double repair, the flat, uninformative leaderboard transformed into a clear spectrum of ability. Suddenly, the models could be distinguished from one another. The most capable models maintained high scores, proving they could handle the difficult tasks when left to their own devices. However, other models that had previously scored highly dropped to zero, revealing that they were entirely dependent on the helper software to function. Some models that looked strong on average were found to be dangerously unreliable, failing completely in specific situations, while smaller, less famous models proved to be surprisingly consistent and robust.

The study also revealed that the reliability of these models varies wildly depending on the specific conditions of the test. A model might look excellent on average but fail completely in its worst-case scenario, a detail that standard tests often hide. By looking at these worst-case outcomes, the researchers found that the ranking of the models changed completely. A model that appeared to be a top performer based on average scores was actually the most fragile, while a smaller model that was previously overlooked proved to be the most reliable.

This work extends beyond a single test. The researchers applied their new audit method to other existing benchmarks and found the same issues. In some cases, the grading system was specific to the test and worked correctly, but the software running the test was still doing too much of the work, making the rankings dependent on which software version was used. In other cases, the grading system itself was flawed, rewarding the wrong things. The researchers concluded that to truly understand an artificial intelligence agent, we must know exactly how much of the work the agent is doing versus how much the test is doing for it, and we must grade the agent on the truth of its answers, not just the neatness of its presentation. Without these changes, the scores we see today may be measuring the test itself rather than the intelligence it claims to evaluate.

Technical Summary: The Double Measurement Confound in Agent Benchmarks

1. Problem Statement

The paper identifies a fundamental flaw in current Large Language Model (LLM) agent benchmarks: benchmark scores often measure the evaluation pipeline rather than the model's capability. The authors argue that reported scores are confounded by two specific artifacts that mask each other, rendering many leaderboards incapable of distinguishing between models of vastly different capabilities.

This "double measurement confound" consists of:

Scaffold Ownership (Artifact 1): Execution-critical decisions (e.g., retrying on failure, pagination, deduplication, filtering summary rows, and final submission) are hardcoded into the evaluation harness (the "scaffold") rather than being made by the model. Consequently, the model merely fills templated slots, and the scaffold performs the actual work.
Scorer Validity (Artifact 2): The evaluators (judges) often grade the shape of the output (formatting, metadata consistency, self-reported logs) rather than comparing the submitted values against a ground truth. This allows fabricated or empty submissions to receive high scores if they are well-formatted, while correct submissions with minor formatting issues are penalized.

The paper demonstrates that under these conditions, a rule-based script with no LLM can achieve scores nearly identical to frontier models (e.g., 96.8 vs. 97.5), and byte-identical outputs are produced by unrelated open models, rendering the benchmark powerless to discriminate model capability.

2. Methodology

The authors propose a measurement-theoretic framework and an executable audit-and-repair protocol called BenchAudit.

Theoretical Framework

The paper formalizes the benchmark score $S(m, c, j)$ as a function of the model $m$ , the scaffold $c$ (harness), and the scorer $j$ .

Non-Identifiability: If the scaffold owns all execution-critical decisions and the scorer grades only shape, the score is invariant to the model choice ( $S(m_1) = S(m_2)$ ) whenever models fill templates equivalently.
Identification: To recover a valid measure of model capability, two conditions must be met simultaneously:
1. De-Scaffolding: Execution-critical decisions must be transferred from the harness to the model.
2. Ground-Truth Scoring: The scorer must evaluate the submitted work product against a deterministic, seeded ground truth, not just its shape.

The Protocol: BenchAudit

The authors introduce BenchAudit, a protocol that:

Transfers Control: Moves decisions (retry, dedup, submission) from the scaffold to the model, creating a "de-scaffolded" agent (Level L2).
Replaces Scoring: Swaps shape-based judges for seeded ground-truth F1 scoring. The ground truth is regenerable from a seed, ensuring deterministic evaluation.
Reports Reliability: Moves beyond mean scores to report worst-case performance, Conditional Value-at-Risk (CVaR), and Reliability@ $\tau$ across multiple seeds. This captures the tail-risk and robustness of the agent.
Generates Validity Cards: Produces a per-benchmark "validity card" that classifies whether a benchmark measures model capability, system performance, or merely self-report compliance.

Experimental Setup: ComtradeBench

The authors instantiate this protocol on ComtradeBench, a seeded benchmark for adversarial, trade-data-shaped tool use.

Task: Agents must fetch paginated data from a mock API, handle faults (429/500 errors, duplicates, drift), deduplicate records, and submit a cleaned dataset.
Adversity: Faults are injected by the environment, not the prompt, making them impossible to "reason away."
Intervention: A $2 \times 2$ experiment compares the original production setup (Scaffold L0 + Shape Scorer) against the repaired setup (Scaffold L2 + Ground Truth Scorer).

3. Key Results

The Flat Leaderboard vs. The Reliability Spectrum

Original Configuration (L0 + Shape Scorer): The leaderboard is nearly flat. Frontier models (Kimi, Claude) and a rule-based script score within 0.7 points of each other. Unrelated open models (Qwen2.5-7B, Llama-3.3-70B) produce byte-identical per-seed reward vectors.
Repaired Configuration (L2 + Ground Truth): The "flat" leaderboard transforms into a graded reliability spectrum.
- Top Performers: Claude-Fable-5 leads with a mean F1 of 0.974 and a worst-case of 0.824.
- Mid-Tier Variance: Models like GPT-5 and GPT-4o show high means (0.930, 0.805) but collapse to 0.00 in worst-case scenarios (bimodal failure).
- Bottom Tier: Llama-3.3-70B and Qwen2.5-7B score 0.000 across all seeds due to "marshaling failures" (submitting empty payloads despite fetching data).

The Necessity of Joint Intervention

A $2 \times 2$ intervention proves that fixing only one artifact is insufficient:

Fixing Scorer Only (L0 + Ground Truth): The scaffold still performs all work; models remain tied.
Fixing Scaffold Only (L2 + Shape Scorer): The shape-based judge compresses the spread, hiding real failures and flooring empty submissions.
Joint Fix (L2 + Ground Truth): Only this combination recovers the full spectrum of model capabilities.

External Validity Audits

$\tau$ -bench: The official scorer is outcome-based (valid), but the scaffold is an uncontrolled axis. Changing the scaffold (e.g., tool-calling vs. ReAct) shifts the same model's reward by up to 0.267 and alters rankings.
BFCL (Berkeley Function-Calling Leaderboard): Uses a specification-based scorer (AST match), which validates spec compliance but not execution success. The scaffold ownership was not probed.
Self-Audit: The authors' own pre-registered stress test (Claim B) showed a significant effect under the original shape-based scorer, but this effect vanished when evaluated against ground truth, confirming the scorer-dependence of the result.

4. Key Contributions

Identification of the Double Confound: The paper formally identifies and proves that scaffold ownership and scorer validity are a joint identification problem. It demonstrates that these artifacts mask each other, making single-sided repairs ineffective.
BenchAudit Protocol: The introduction of an executable audit pipeline that transfers execution decisions to the model, scores against ground truth, and reports tail-risk metrics (CVaR, worst-case).
ComtradeBench & Validity Cards: The release of a seeded generator-based benchmark and a "validity card" framework that categorizes benchmarks by what they actually measure (e.g., model capability vs. system performance vs. self-report compliance).
Empirical Evidence: The demonstration that removing the confound reorders model rankings, revealing that "high mean" models can have catastrophic worst-case reliability, while smaller models can be more robust.

5. Significance and Claims

The paper claims that benchmark scores should not be interpreted as evidence of model capability unless the benchmark passes specific validity checks.

Reframing Evaluation: The authors argue that current leaderboards often measure the scaffold's robustness rather than the model's. A score is only interpretable as a property of the model if the model owns the execution-critical decisions and the scorer is criterion-valid (ground-truth based).
Reliability over Mean: The paper asserts that worst-case reliability is a critical metric that can reorder the frontier. A model with a high mean but a worst-case of 0.00 is less reliable than a model with a slightly lower mean but a high worst-case floor.
Practical Framework: The proposed protocol is designed to be applied on top of existing benchmarks to diagnose their validity, rather than replacing them entirely. It provides a practical path to "de-scaffold" and repair existing evaluations.
Modesty: The authors acknowledge limitations, noting that ComtradeBench is a stylized domain (trade data extraction) and that the "de-scaffolding" intervention may not be trivial to implement for all benchmark types (e.g., those requiring complex environment interaction). They also note that some benchmarks (like BFCL) may not be amenable to ground-truth scoring if no ground truth exists, limiting the scope of the "model capability" claim to specific task types.

In conclusion, the paper provides a rigorous measurement-theoretic argument that without controlling for scaffold ownership and scorer validity, agent benchmarks are measuring the wrong thing, and offers a concrete, reproducible protocol to fix this.