🤖 AI

PaperRepro: Automated Computational Reproducibility Assessment for Social Science Papers

PaperRepro is a novel two-stage, multi-agent framework that automates computational reproducibility assessment for social science papers by separating code execution from evaluation, achieving state-of-the-art performance on the REPRO-Bench benchmark through enhanced context handling, task-specific tooling, and explicit result capture.

Original authors: Linhao Zhang, Tong Xia, Jinghua Piao, Lizhen Cui, Yong Li

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Linhao Zhang, Tong Xia, Jinghua Piao, Lizhen Cui, Yong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a food critic reviewing a new cookbook. The author claims their "Secret Chocolate Cake" is the best in the world. But to truly trust the review, you need to bake the cake yourself using the author's recipe and ingredients to see if it actually tastes as good as they say.

In the world of science, this is called reproducibility. If a scientist publishes a paper with cool findings, other scientists need to be able to run the same "recipe" (code and data) to get the same results. If they can't, the science might be flawed.

The problem? Checking these recipes is a nightmare. It's like trying to bake a cake in a dark kitchen where the instructions are written in a foreign language, the oven is broken, and the ingredients are hidden in boxes you have to guess are in. Doing this manually takes human experts months or even years.

Enter PaperRepro. Think of PaperRepro as a team of super-efficient, robot kitchen assistants designed to automate this baking process.

The Problem: Why Humans (and Old Robots) Struggle

The authors explain that previous attempts to automate this failed for three main reasons:

Memory Overload: The robots tried to read the entire cookbook, the code, the error logs, and the final cake all at once. Their "brains" (computer memory) got full, and they forgot the important details.
Wrong Tools: They tried to use a generic screwdriver to fix a delicate watch. They didn't have the right tools to extract specific charts or figures from a PDF document to compare them.
Missing the Result: Sometimes the robot would run the code, but the "cake" (the result) would only appear on a screen for a split second and then vanish. The robot would finish the job but have no proof of what it made.

The Solution: PaperRepro's Two-Stage Kitchen

PaperRepro solves this by splitting the job into two distinct teams (stages) and giving them specialized tools.

Stage 1: The "Bakers" (Execution Stage)

This team is responsible for actually running the code.

The Planner (Setup Agent): Before baking, this robot reads the messy instructions, figures out which ingredients go in first, and sets up the kitchen. It fixes broken paths (like changing "Use my specific oven" to "Use the oven in this room").
The Baker (Execution Agent): This robot follows the plan. If the recipe says "save the cake to the counter," but the original code just shows it on a screen, this robot edits the code to make sure the cake is actually saved as a file. If something breaks, it tries to fix it, but it keeps a copy of the original recipe so it doesn't destroy the evidence.
The Goal: To produce a physical "artifact" (a saved file) of the result, no matter how messy the original process was.

Stage 2: The "Judges" (Evaluation Stage)

Once the Bakers have produced the cake, the Judges step in.

The Taster (Scoring Agent): This robot compares the cake the Baker made with the picture in the cookbook. It uses special tools to "see" the images and tables clearly. It doesn't just guess; it looks at the evidence files saved by the Bakers.
The Reporter (Report Agent): This robot writes the final review. It creates a clear, structured report saying, "The cake looks 90% like the picture, but the frosting is slightly off," and gives it a score.

Why This Works Better

The magic of PaperRepro is separation of duties.

The Bakers don't have to worry about grading; they just focus on making the cake.
The Judges don't have to worry about baking; they just focus on comparing.
They use specialized tools (like a magnifying glass for charts) that previous robots didn't have.
Everything is saved to files, so nothing is lost in the "air" of a computer screen.

The Results

The researchers tested PaperRepro on a benchmark called REPRO-Bench, which is like a standardized test of 112 different "recipes" from social science papers.

PaperRepro got a score of 44.6% accuracy.
The previous best robot only got 36.6%.
That might not sound like a huge jump, but in the world of AI, a 21.9% relative improvement is massive. It means PaperRepro is significantly better at figuring out if a scientific paper is trustworthy.

They also realized the test itself had some errors (like a test question with the wrong answer key), so they fixed the test and created a harder version called REPRO-Bench-S to make sure future robots are truly tough enough for the job.

The Big Picture

PaperRepro is a step toward a future where we don't have to wait years for humans to verify scientific claims. It's like having a tireless, super-smart intern who can check the math and the code for thousands of papers, ensuring that the science we rely on for policy and decisions is actually real.

In short: PaperRepro is a team of specialized robot assistants that first fix and run the messy code, then carefully compare the results to the original paper, giving us a reliable score on whether the science holds up.

1. Problem Definition

The paper addresses the critical challenge of computational reproducibility in social science research. While reproducibility is essential for scientific credibility, manual assessment is costly, time-consuming, and prone to human error.

Context: Social science findings often rely on complex workflows involving multiple scripts (e.g., R, Stata), implicit dependencies, and specific software environments.
Challenges for Automation: Existing automated approaches (often based on Large Language Models/Agents) fail due to three main gaps:
1. Context Bottleneck: Agents cannot track the vast number of files, logs, and intermediate outputs required for social science reproduction.
2. Task-Specific Tooling Gap: Generic agents lack tools to extract specific tables/figures from PDFs or handle domain-specific execution flows.
3. Result Capture Gap: Agents often run scripts but fail to capture the reproduced results (which may appear only in GUIs or logs) for comparison against the paper's reported results.

2. Methodology: PaperRepro

The authors propose PaperRepro, a novel two-stage, multi-agent framework designed to separate execution from evaluation. This architecture mitigates context limits and ensures traceability.

A. Architecture Overview

The system processes a paper ( $P$ ), its reproduction package ( $C$ ), and target items ( $I$ ) through two distinct stages:

Artifact-Driven Execution Stage: Focuses on running the code and capturing results.
Evidence-Grounded Evaluation Stage: Focuses on comparing captured artifacts with the paper to assign a score.

B. The Execution Stage

This stage is decomposed into two specialized agents:

Setup Agent:
- Role: Analyzes the repository (README, code) to identify entry points and dependencies.
- Action: Constructs an executable reproduction plan ( $D_p$ ) and generates a unified installation script to handle environment setup.
Execution Agent:
- Role: Follows the plan to run scripts.
- Key Innovation: It performs minimal, semantics-guided edits to the code (e.g., changing hard-coded paths, adding save commands) to ensure results are persisted as explicit files (artifacts) rather than just appearing in logs or GUIs.
- Output: Produces reproduced artifacts ( $A_r$ ) and an execution summary ( $D_e$ ) containing logs and code quality assessments.

C. The Evaluation Stage

This stage uses the captured artifacts to assess reproducibility:

Scoring Agent:
- Role: Compares the paper's reported results with the reproduced artifacts.
- Tools: Uses multimodal tools (element extractors, image converters) to visualize tables and figures from the PDF and the reproduced outputs.
- Action: Assigns a reproducibility score (1–4) based on consistency and code quality.
Report Agent:
- Role: Aggregates execution traces and scoring outcomes into a structured, human-readable report.

D. Key Design Principles

File Persistence: All critical outputs (plans, logs, modified code, results) are saved as files to ensure traceability and allow human review.
Context Control: Uses task decomposition, paginated file access, and log truncation to prevent context overflow.
Expert Prompting: Agents are equipped with domain-specific prompts and few-shot examples (e.g., how to handle R/Stata ecosystems).
Failure Recovery: Includes mechanisms for non-intrusive code modification (creating copies rather than overwriting) and robust continuation if upstream steps fail.

3. Key Contributions

Novel Multi-Agent Framework: Introduced a two-stage pipeline that decouples execution from evaluation, significantly improving reliability in complex social science workflows.
Specialized Tooling: Developed task-specific tools (e.g., PDF element extraction, image conversion, minimal code editing) that address the specific needs of reproducibility assessment.
Benchmark Refinement: Identified annotation errors in the existing REPRO-Bench and introduced REPRO-Bench-S, a difficulty-stratified benchmark that allows for more diagnostic evaluation of automated systems.
State-of-the-Art Performance: Demonstrated that PaperRepro significantly outperforms existing baselines in accuracy and executability.

4. Experimental Results

The system was evaluated on REPRO-Bench (112 social science paper instances) and the refined REPRO-Bench-S.

Performance Metrics:
- Accuracy (Score Agreement): PaperRepro achieved 44.6% accuracy on REPRO-Bench, a 21.9% relative improvement over the strongest baseline (REPRO-Agent at 36.6%). On the refined REPRO-Bench-S, accuracy rose to 50.9%.
- Applicability: Achieved 100% (produced valid outputs for all instances), whereas baselines often failed to generate valid files.
- Executability: Achieved 65.2% (successfully ran the package and captured artifacts), outperforming baselines like AutoGPT (45.7%).
Failure Analysis: The system struggles most with "Level-3" (hard) instances involving complex dependencies or unclear instructions. Common failure modes include setup failures (missing dependencies) and planning failures (incorrect script ordering).
Case Study: Successfully reproduced a complex Nature paper figure involving 6 R scripts and cross-script dependencies in ~15 minutes, a task taking a human expert ~1 hour.

5. Significance and Impact

Scalability: PaperRepro offers a scalable solution to the growing burden of verifying social science research, which is critical as AI-generated research increases.
Reliability: By separating execution from evaluation and enforcing file persistence, the system provides a more transparent and auditable assessment process.
Benchmark Advancement: The introduction of REPRO-Bench-S sets a new standard for evaluating reproducibility tools, moving beyond simple pass/fail metrics to diagnostic evaluation based on execution difficulty.
Future Direction: The work paves the way for automated pre-publication checks and continuous monitoring of scientific integrity across various domains.

In summary, PaperRepro represents a significant leap forward in automating scientific verification by combining multi-agent collaboration, specialized tooling, and a rigorous two-stage workflow to tackle the inherent complexity of social science code reproduction.