Technical Summary: CORE-Bench

Problem Statement

Computational reproducibility—the ability to reproduce scientific study results using provided code and data—is a cornerstone of the scientific process. However, despite the widespread availability of data and code, a significant proportion of studies across diverse fields (including computer science, medicine, economics, and psychology) fail to be computationally reproducible. Challenges include unspecified library versions, architectural differences (e.g., ARM vs. x86), operating system incompatibilities, and inherent result variance.

While recent advancements in AI agents and compound AI systems have shown promise in automating complex coding tasks (e.g., SWE-bench), there is a lack of benchmarks designed to evaluate agents on the specific, routine, yet critical task of reproducing existing scientific research. Before agents can be trusted to conduct novel research, they must first demonstrate the ability to faithfully reproduce existing work.

Methodology

Benchmark Construction (CORE-Bench)

The authors introduce CORE-Bench (Computational Reproducibility Agent Benchmark), a dataset comprising 270 tasks derived from 90 scientific papers across three disciplines: Computer Science, Social Science, and Medicine. The papers were sourced from CodeOcean, a platform known for hosting reproducible research capsules, ensuring that a ground truth for successful reproduction exists.

Selection Criteria:
The authors filtered 5,090 CodeOcean capsules using ten criteria to ensure feasibility and construct validity (Table 2). Key criteria included:

• Correspondence to a publicly accessible paper.
• Implementation in Python or R.
• Presence of a README file.
• Execution time under 45 minutes on standard hardware.
• Low variance in results (ensuring deterministic or near-deterministic outputs).
• Size under 10 GB.

Task Difficulty Levels:
To evaluate a spectrum of agent capabilities, the authors created three difficulty levels for each of the 90 papers (Table 3):

1. CORE-Bench-Easy: The agent is provided with the complete code output from a successful run. The task is purely information extraction from existing text or figures.
2. CORE-Bench-Medium: The agent is provided with a Dockerfile and text instructions. The agent must execute the Docker command to reproduce the code and then extract results.
3. CORE-Bench-Hard: The agent is provided only with the README file. It must independently install dependencies, determine the correct execution command, run the code, and extract results. This level mimics the real-world scenario of reproducing a paper without pre-packaged containers.

Evaluation Metrics:

• Accuracy: A task is marked correct only if the agent answers all associated questions correctly. Questions require extracting specific values (e.g., model accuracy, axis labels, correlation coefficients) from terminal outputs, PDFs, HTML, or figures.
• Vision-Language Integration: Tasks include both text-based questions and vision-based questions requiring the analysis of figures and plots.
• Evaluation Harness: To ensure fairness and parallelizability, the authors built an evaluation harness that runs each agent-task pair in an isolated Virtual Machine (VM). This prevents agents from tampering with the benchmark and allows hundreds of tasks to run in parallel, reducing evaluation time from ~20 days (sequential) to ~2 hours.

Baseline Agents

The authors evaluated two agent architectures using GPT-4o and GPT-4o-mini as the underlying LLMs:

1. AutoGPT: A general-purpose agent with minimal modifications (addition of a query_vision_language_model tool).
2. CORE-Agent: A task-specific variant of AutoGPT customized for each difficulty level. Modifications included:
   • Programmatic checks to ensure the submission of a correctly formatted report.json.
   • Specific prompting hints (e.g., using pdftotext for PDFs, prioritizing specific files, using absolute paths instead of environment variables, and pre-installing dependencies before running code).

Key Results

Overall Performance:

• CORE-Agent (GPT-4o) achieved the highest performance: 60.00% on Easy, 57.78% on Medium, and 21.48% on Hard tasks.
• AutoGPT (GPT-4o) performed significantly lower, particularly on Hard tasks (6.67%), highlighting the necessity of task-specific adaptations.
• GPT-4o-mini showed lower accuracy across all agents and difficulty levels but demonstrated a favorable cost-accuracy trade-off.

Key Observations:

1. Impact of Task-Specific Modifications: Adapting a generalist agent (AutoGPT) with specific prompts and constraints (CORE-Agent) yielded massive improvements, especially for weaker models (e.g., GPT-4o-mini improved from 8.89% to 44.44% on Easy tasks).
2. Difficulty Gradient: Performance dropped significantly as difficulty increased, confirming that dependency management and command execution (Hard level) remain major bottlenecks.
3. Modality Differences: Agents performed significantly better on text-based questions (87.88% on Easy) than vision-based questions (59.26% on Easy), often struggling to identify relevant figures among multiple outputs.
4. Language Differences: Python tasks were significantly easier to reproduce than R tasks, partly due to the complexity of parsing R-generated PDF manuscripts and longer dependency installation times.
5. Failure Modes: Common failures included:
   • Retrieving information from the wrong figure or file.
   • Ignoring Docker instructions and attempting manual reproduction (leading to context limits).
   • Getting stuck in dependency installation loops (Hard level).
   • Attempting to access external websites (e.g., CodeOcean) that require JavaScript rendering, which agents could not handle.

Significance and Claims

The paper posits that reproducing existing research is a necessary prerequisite for building agents capable of conducting novel scientific research. The authors claim that:

1. Feasibility of Automation: While full automation of computational reproducibility is currently difficult (21% accuracy on the hardest tasks), it is not impossible. Simple, task-specific modifications to generalist agents can yield substantial performance gains.
2. Utility of CORE-Bench: The benchmark provides a standardized, realistic, and challenging environment to measure agent progress. It covers a diverse set of skills including coding, shell interaction, retrieval, and vision-language interpretation.
3. Real-World Application: Successful agents could assist authors in verifying reproducibility before publication, help independent researchers replicate studies, and aid journal editors in assessing submissions.
4. Safety and Guardrails: The study highlights the need for better guardrails to prevent agents from taking unsafe or unexpected actions (e.g., creating thousands of accounts or failing to render JavaScript-dependent sites) when deployed in real-world scientific workflows.

The authors conclude that while the current state of agents leaves "vast scope for improvement," CORE-Bench serves as a foundational step toward spurring the development of reliable research agents and improving the overall state of computational reproducibility in science.