Technical Summary: ReproRepo

Problem Statement

Reproducing research results is fundamental to scientific progress, yet verifying findings and building upon prior work often requires navigating complex combinations of papers, code, data, and experimental artifacts. While Large Language Model (LLM) agents show promise in automating aspects of reproducibility, existing evaluation benchmarks face a critical scalability bottleneck. Prior approaches rely heavily on substantial manual effort for data curation, expert-curated questions, artifact preparation, and manual evaluation. Consequently, these datasets typically cover fewer than 100 papers, often feature only high-quality artifacts, and struggle to remain relevant in the fast-changing research landscape due to the difficulty of updating them.

Methodology: The ReproRepo Framework

To address scalability, the authors introduce ReproRepo, a framework for scalable, realistic, and easily updatable evaluation of LLM agents on reproducibility auditing. The core innovation is leveraging GitHub issues as naturally occurring human supervision signals. Instead of experts manually designing tasks or injecting errors, ReproRepo treats user-reported issues as evidence of real-world reproduction blockers.

Data Construction

The framework constructs a benchmark from three primary sources:

1. Conference Papers: Accepted papers from major machine learning conferences (NeurIPS 2022/2024, ICLR 2026).
2. GitHub Repositories: Linked codebases associated with these papers.
3. GitHub Issues: Both open and closed issues reported by users attempting to reproduce the work.

The data pipeline involves filtering papers to retain only those with usable repositories and at least one reproducibility-related issue. For evaluation, the system builds specific repository snapshots:

• Open Issues: The repository state at the time of collection.
• Closed Issues: The repository state immediately before the issue was created (pre-fix), ensuring agents evaluate the code as a user would have seen it when encountering the problem.
• Post-Fix Snapshots: For closed issues with patches, a post-fix state is recovered to serve as a validation set for measuring false positives (i.e., ensuring agents do not report issues that have already been resolved).

Agent Task and Protocol

The evaluation focuses on a compute-light static inspection setting. Agents are provided with the paper PDF and a repository snapshot but are strictly prohibited from:

• Executing code or setting up environments.
• Accessing the internet, external APIs, or hidden benchmark metadata.
• Viewing the actual GitHub issues used for ground truth.

Agents are tasked with performing a static audit to predict potential reproducibility blockers, outputting a ranked list of findings ordered by the estimated likelihood of a user encountering them.

Evaluation Metrics

Agent predictions are assessed by a separate LLM judge against the hidden human-reported issues. The assessment uses three labels:

• Exact Match (EM): The finding identifies the same practical problem and trigger.
• Semantic Match (SM): The finding identifies a semantically related blocker with concrete overlap but incomplete specificity (e.g., same pipeline but different failure point).
• None: No useful evidence that the agent discovered the issue.

Metrics are aggregated at the issue level (fraction of hidden issues recovered) and the paper level (fraction of papers where at least one or all hidden issues are recovered). A False Positive Rate (FPR) is calculated by running agents on post-fix snapshots; if an agent reports a finding matching a fixed issue, it is counted as a false positive.

Key Contributions

1. Scalable Framework: Introduction of ReproRepo, which uses GitHub issues as free, naturally occurring supervision, enabling diverse and continuously updatable benchmarks with minimal manual annotation.
2. Large-Scale Benchmark: Construction of a dataset comprising 1,149 recent machine learning papers and 7,553 human-reported reproduction blockers, significantly larger than prior reproducibility benchmarks.
3. Systematic Evaluation: A comprehensive evaluation of frontier LLM agents (including Claude Code and Codex paired with models like GPT-5.5 and Claude Opus 4.7) under static inspection constraints.

Experimental Results

The study evaluated four frontier model-agent configurations on 1,149 papers. Key findings include:

• High Recovery of Blockers: Surprisingly, static agents (without code execution) can identify a substantial fraction of human-reported blockers. The best configuration (Codex with GPT-5.5) surfaced at least one semantically related human-reported blocker for ~90% of papers (89.7% for ICLR 2026, 93.3% for NeurIPS 2024 main).
• Semantic vs. Exact Localization: While agents are highly effective at surfacing visible failures and identifying the correct semantic region (Semantic Match rates are significantly higher than Exact Match rates), they struggle with exact localization. For instance, on ICLR 2026, the Semantic Match rate was 58.1% while the Exact Match rate was 25.4%.
• Low False Positive Rate: The agents demonstrated high precision, with False Positive Rates remaining low (maximum 3.50% across models), indicating that reported issues are grounded in plausible static evidence rather than hallucinations.
• Failure Categories: Agents performed best on "Silent Wrong Setup" (e.g., missing artifacts, mismatched configurations) and "Crash Immediate" (e.g., broken imports). They struggled more with "Crash Late" and "Silent Wrong Number," which often require execution or comparison against reported metrics.
• Cost-Performance Trade-off: While GPT-5.5 achieved the highest performance, DeepSeek-V4-Pro achieved similar paper-level semantic match rates (89.0% vs. 89.7%) at a fraction of the cost (23% per run), suggesting a strong trade-off for broad triage.

Significance and Claims

The paper claims that ReproRepo provides a reusable, scalable framework for future evaluations of LLM agents on real-world reproducibility auditing. The authors assert that:

• Static inspection is viable for triage: Agents can effectively identify real-world reproducibility problems without executing code, making them useful for early-stage review and author triage.
• GitHub issues are a valid supervision signal: Using naturally occurring user reports allows for the construction of large-scale, realistic benchmarks that are difficult to maintain with manual expert curation.
• Limitations remain: The authors modestly note that while agents are good at surfacing visible failures, they are not yet sufficient for exact localization of errors or identifying less visible failure types that require execution. The framework is intended as a triage signal rather than a definitive judgment on reproducibility.

The work concludes that while current agents cannot fully replace execution-based validation, they offer a powerful tool for scaling the initial audit of research reproducibility.