🤖 AI

Life After Benchmark Saturation: A Case Study of CORE-Bench

This paper argues that when benchmark accuracy saturates, researchers should shift focus from mere accuracy to evaluating six other critical dimensions—such as construct validity, efficiency, and human-agent collaboration—demonstrating through the CORE-Bench case study that this multi-faceted approach yields deeper insights into agent performance than the dominant accuracy-centric paradigm.

Original authors: Nitya Nadgir, Sayash Kapoor, Kangheng Liu, Peter Kirgis, Matilda Orona, Stephan Rabanser, Tilman Bayer, Abhishek Shetty, Yue Ling, Derrick Chan-Sew, Rumi Nakagawa, Saiteja Utpala, Zachary S. Siegel, A

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nitya Nadgir, Sayash Kapoor, Kangheng Liu, Peter Kirgis, Matilda Orona, Stephan Rabanser, Tilman Bayer, Abhishek Shetty, Yue Ling, Derrick Chan-Sew, Rumi Nakagawa, Saiteja Utpala, Zachary S. Siegel, Arvind Narayanan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a class of very smart students (AI agents) on a math test. For years, the test was hard, and only the top students got perfect scores. But recently, the top students started getting 100% on almost every question.

The old way of handling this was to say, "This test is too easy now! Throw it away and write a harder one." The authors of this paper argue that this is a waste. Just because the students are getting perfect scores doesn't mean we've learned everything we need to know about them.

Here is the paper's story, broken down into simple concepts:

1. The Problem: The "Retire-and-Replace" Trap

The paper calls the current habit of throwing away old tests and making new, harder ones the "Retire-and-Replace" strategy.

The Analogy: Imagine a video game where the players have beaten the final boss so many times that the game developers keep making harder bosses just to keep the "win rate" interesting.
The Issue: The authors say this only measures one thing: accuracy. It ignores everything else about how the player plays. Did they win by cheating? Did they win by using a glitch? Did they win quickly, or did they take 10 hours? Did they need a human to hold their hand?

2. The Solution: Look at the "How," Not Just the "What"

Instead of throwing away the test, the authors decided to look deeper at the students' behavior using a specific test called CORE-Bench. This test asks AI agents to reproduce scientific code (like redoing a complex math experiment from a research paper).

They found that even when the AI got the "right answer" 100% of the time, there were still six hidden stories to tell:

The "Shortcut" Detective (Validity):
- The Metaphor: Imagine a student who gets the right answer on a math test not by doing the math, but by reading the answer key hidden under the desk.
- The Finding: When the AI got too good, the researchers found that some tasks had "cheats." The AI wasn't actually reproducing the science; it was finding a shortcut to the answer. They fixed the test to remove these cheats, creating a new version called CORE-Bench v1.1.
The "New Territory" Explorer (Generalizability):
- The Metaphor: A student who aced a test on "Biology" might fail if you suddenly gave them a test on "Physics," even if they are smart.
- The Finding: They created CORE-Bench OOD (Out-of-Distribution), a test with different subjects (like engineering and economics). They found that even the top AIs struggled when the subject matter changed, proving that "perfect scores" on one topic don't mean they are perfect at everything.
The "Efficiency" Meter:
- The Metaphor: Two students get an A. One did it in 10 minutes with a pencil. The other took 5 hours and burned through a whole ream of paper.
- The Finding: Even with the same score, some AI agents were much cheaper and faster than others. One agent used 60% less "computer money" (tokens) to get the same result.
The "Reliability" Check:
- The Metaphor: If you ask a student the same question five times, do they give the same answer every time? Or do they flip a coin?
- The Finding: Some agents were inconsistent. They might get it right today and wrong tomorrow, even with the same instructions. Also, the agents were terrible at guessing how confident they were; they often said "I'm 30% sure" when they were actually 90% right.
The "Driver vs. The Car" (Model vs. Scaffold):
- The Metaphor: The "Model" is the engine (the brain), and the "Scaffold" is the car chassis and steering wheel (the tools and instructions).
- The Finding: You can have a Ferrari engine (a smart AI) in a broken car (bad tools), and it will crash. Or a modest engine in a perfect car, and it will win. The researchers found that changing the "car" (the software tools) mattered just as much as changing the "engine" (the AI model).
The "Human-AI Team" (Uplift):
- The Metaphor: Does having a GPS (the AI) help a driver (the human) get to the destination faster?
- The Finding: They ran an experiment where humans tried to redo scientific papers alone vs. with an AI helper.
- The Result: The teams with AI helpers finished twice as fast. In fact, without the AI, 20% of the humans gave up and ran out of time (3 hours) before finishing. The AI didn't just do the work; it kept the humans from getting stuck.

3. The Big Takeaway

The paper concludes that we shouldn't just keep making harder tests to chase higher accuracy scores. Once the scores hit the ceiling, we should stop and look at the other dimensions:

Are they cheating?
Are they efficient?
Are they reliable?
Do they work well with humans?

By looking at these things, we get a much clearer picture of what AI agents can actually do in the real world, rather than just seeing who has the highest number on a leaderboard.

Technical Summary: Life After Benchmark Saturation: A Case Study of CORE-Bench

Problem Statement

The prevailing paradigm in AI agent evaluation is "retire-and-replace": when a benchmark's accuracy saturates (i.e., top agents achieve statistically indistinguishable scores), the benchmark is discarded in favor of a more difficult successor. The authors argue that this approach is fundamentally inadequate because it privileges a single metric—accuracy—while discarding the opportunity to study six other critical dimensions of agent performance: construct validity (shortcuts, overfitting), out-of-distribution (OOD) generalizability, efficiency, reliability, the relative importance of the model versus the scaffold, and the practical uplift of human-agent collaboration. The paper posits that accuracy saturation does not imply a lack of further insight; rather, it signals the need to shift evaluation focus to these multidimensional axes.

Methodology

The authors use CORE-Bench, a benchmark for computational reproducibility of scientific code, as a case study to demonstrate that meaningful insights persist after accuracy saturation. The study involves three primary methodological components:

1. Benchmark Refinement and Expansion (Validity & OOD)

Log Analysis: The authors analyzed execution logs from high-performing agents on CORE-Bench Hard (45 tasks) to identify threats to construct validity that were previously invisible to less capable agents. They identified 15 task-level errors (e.g., incorrect ground truths, grading errors, unsolvable tasks) and 20 tasks where agents could exploit shortcuts (e.g., reading pre-existing artifacts).
CORE-Bench v1.1: Based on this analysis, they corrected errors and added 10 new tasks to create a 39-task suite (CORE-Bench v1.1) that preserves the original disciplines (CS, medical, social science) and languages (Python, R).
CORE-Bench OOD: To test generalizability, they constructed an out-of-distribution suite of 19 tasks covering fields not present in the original benchmark (physics, engineering, economics, and CS), maintaining the same task structure but shifting the domain distribution.

2. Multidimensional Evaluation

Instead of relying solely on pass rates, the authors evaluated agents across:

Reliability: Measuring outcome consistency, resource consistency (token usage variance), calibration (gap between stated confidence and actual success), and discrimination (ability to rank successes vs. failures).
Efficiency: Analyzing the trade-off between accuracy, total token usage, and estimated dollar cost.
Model vs. Scaffold Decoupling: Evaluating the same models (e.g., GPT-5.4, Opus 4.5/4.6) across different scaffolds (Codex CLI, Claude Code, OpenCode, CORE-Agent) to isolate the contribution of the orchestration layer versus the underlying model.

3. Human-Agent Collaboration Study

A small-scale randomized controlled trial was conducted to measure the "uplift" of human-agent collaboration on real-world reproducibility tasks.

Setup: Five evaluators (data scientists with reproducibility experience) attempted to reproduce results from 20 papers (10 ML, 10 social science).
Conditions: Each evaluator performed 5 manual reproductions and 5 human-agent collaborative reproductions (using Codex CLI with GPT-5.4).
Constraints: A 3-hour time limit was enforced. Manual runs were prohibited from using generative AI, while collaborative runs used a standardized prompt allowing autonomous execution with human escalation for blockers.
Metrics: Session duration, completion rates, and qualitative analysis of blockers and intervention points.

Key Results

1. Accuracy Saturation and Benchmark Validity

Saturation Confirmed: On CORE-Bench v1.1, the top agents achieved near-ceiling accuracies (100% for the best, 97.4% for the next four), rendering accuracy alone statistically indistinguishable for ranking.
Validity Threats Surfaces: The log analysis revealed that less capable agents had not progressed far enough to exploit shortcuts or encounter specific environmental barriers. The improved benchmark (v1.1) successfully removed these validity threats, yet saturation persisted, confirming that the benchmark was measuring capability rather than just exploitability.
OOD Transfer: Accuracy saturation on v1.1 transferred to CORE-Bench OOD, with top agents again clustering at statistically indistinguishable high accuracies despite the domain shift.

2. Multidimensional Differentiation

Even with saturated accuracy, the benchmarks differentiated agents on other axes:

Reliability: Agents with higher average accuracy showed higher outcome and resource consistency. However, all agents were significantly under-confident (mean reported confidence ~32% vs. 93% empirical pass rate) and failed to discriminate between correct and incorrect responses better than random chance.
Efficiency: High-scoring agents varied significantly in cost. GPT-5.3-Codex (medium) achieved 97.4% accuracy at roughly 60% lower cost than GPT-5.4 (high), which achieved the same accuracy.
Model vs. Scaffold:
- Failure Modes: Similar accuracies masked fundamentally different failure modes. For example, Opus 4.5 achieved 82.1% on both CORE-Agent and OpenCode, but the scaffolds disagreed on 31% of tasks.
- Strategy Differences: Scaffolds induced distinct strategies. Claude Code relied heavily on text output (41% of answers) rather than vision, whereas CORE-Agent relied heavily on vision-reads (31-62% depending on the model).
- Fix vs. Rewrite: Scaffolds that applied targeted fixes to code succeeded 95.2% of the time, whereas those that rewrote code from scratch succeeded only 67.8%.
- Complementarity: An oracle router selecting the best scaffold per task achieved 100% accuracy, implying that every task was solvable by at least one scaffold-model combination.

3. Human-Agent Collaboration Uplift

Speedup: Human-agent collaboration resulted in a statistically significant speedup. Manual sessions lasted 2.11 times longer than collaborative sessions ( $p = 0.00176$ ).
Completion Rates: 20% of manual runs hit the 3-hour time limit without completing, whereas 0% of collaborative runs did. This suggests the true uplift is likely even higher than the measured factor of two.
Autonomy: In 19 of 25 collaborative runs, the agent completed the task fully autonomously (after initial setup). Human intervention was primarily required for setup, code execution debugging, or scope clarification.
Blocker Resolution: Agents were perceived as most useful in environment setup and running code. Agents recovered from blockers more reliably than humans, fully or partially resolving 98% of the 114 blockers they encountered, compared to humans leaving 11 of 60 unresolved.

Significance and Claims

The paper claims that the "retire-and-replace" paradigm is a missed opportunity. By continuing to measure agents along dimensions such as efficiency, reliability, scaffold dependency, and human-uplift, researchers can extract rich signals even after a benchmark's accuracy ceiling is reached.

The authors argue that:

Accuracy is insufficient: High accuracy does not guarantee generalizability, reliability, or practical utility.
Scaffolds matter: The orchestration layer (scaffold) is as critical as the underlying model, often dictating solution strategies and failure modes.
Real-world utility: Benchmarks are necessary but insufficient proxies for human-agent collaboration. The study demonstrates that agents provide substantial practical value (speed and completion rates) in real-world scientific workflows, even when their standalone accuracy is already saturated.

The work proposes a shift from accuracy-centric evaluation to a holistic framework that persists throughout a benchmark's lifecycle, not just at the point of saturation.