ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review

Technical Summary: Agentic Reproducibility Assessment (ARA)

Problem Statement
Scientific peer review is increasingly unable to assess reproducibility at the scale and complexity of modern research output. The exponential growth in publications, accelerated by large language models (LLMs) as writing assistants, has strained traditional quality assurance mechanisms. While reproducibility is a necessary condition for validating scientific contributions, assessing it requires reconstructing experimental dependencies, methodological choices, data flows, and result-generating procedures. This process often exceeds the capacity of human reviewers given time constraints and the volume of submissions. Existing automated approaches are often domain-specific, rely on simple checklist-based parsing, or require access to external replication packages (code, data) beyond the publication text itself. Consequently, reproducibility assessments remain underrepresented in standard workflows and difficult to scale.

Methodology
The paper introduces Agentic Reproducibility Assessment (ARA), a framework that formalizes reproducibility assessment as a structured reasoning task executable by LLM-based agent systems directly from scientific documents. ARA operates without requiring external artifacts, source code, or execution environments. The pipeline consists of three primary stages:

1. Workflow Graph Construction:
   Given a scientific paper $D$, ARA constructs a directed workflow graph $G = (V, E)$. The nodes $V$ are partitioned into four domain-agnostic stages:

   • Sources: Datasets, assumptions, and research questions.
   • Methods: Intermediate transformations, algorithms, parameters, and software/hardware resources.
   • Experiments: Evaluation protocols, baselines, runtime conditions, and configurations.
   • Sinks: Result-relevant figures, tables, and reported conclusions.
     Edges $E$ represent dependency relations between these components. An LLM agent extracts these components using structured queries, grounding responses in literal source quotes to ensure traceability.

2. Micro-Level Scoring:
   The reconstructability of individual nodes is assessed using an ordinal scoring function $r(\cdot) \in \{1, 2, 3, 4\}$, where 1 denotes missing information and 4 denotes sufficient detail for independent reconstruction. This scoring is performed stage-specifically, distinguishing between aspects like algorithm clarity (for methods) versus dataset availability (for sources).

3. Reproducibility Scoring:
   Micro-level assessments are aggregated into a final composite reproducibility score $R$ via two orthogonal axes:

   • Content Score ($R_c$): Measures the descriptive sufficiency of nodes (sources, methods, experiments, sinks) for independent reconstruction. It is a weighted convex combination of stage-specific average scores.
   • Structural Score ($R_s$): Measures the quality of the workflow graph's connectivity. It penalizes disconnected sources, orphan sinks, unresolved references, broken source-to-sink paths, and graph fragmentation.
     The final score is the geometric mean: $R = \sqrt{R_c \cdot R_s}$. This formulation ensures that weakness in either descriptive detail or structural connectivity cannot be compensated by strength in the other.

Key Contributions

• Formalization of Document-Level Reasoning: ARA models reproducibility assessment not as a checklist or a full execution task, but as a structured reasoning problem over the evidence-generation workflow extracted from text.
• Domain-Agnostic Framework: Unlike prior work restricted to specific fields (e.g., computer science or social sciences), ARA is designed to operate across heterogeneous scientific domains.
• Scalable Diagnostic Support: The system provides a scalable mechanism to identify missing reporting elements, incomplete workflows, and reproducibility bottlenecks, intended to complement rather than replace human peer review.
• Comprehensive Benchmarking: The authors evaluate ARA on the ReScience C dataset, comprising 213 human-validated computational reproducibility studies across multiple disciplines. This represents the largest cross-domain benchmark of its kind to date.

Results
Experiments were conducted across 213 ReScience C articles, comparing ARA's assessments against human-verified reproduction workflows and reporting structures.

• Consistency: The framework demonstrates substantial stability across different LLM architectures (including Gemini, GPT-4.1, and Qwen variants), sampling temperatures, and repeated runs. Graph structural consistency and reproducibility score variability remain low, indicating the system is not driven by hallucination or oracle-like randomness.
• Performance: ARA achieves consistent agreement with human assessments, reaching approximately 61% accuracy across three reproducibility assessment benchmarks (ReScience C, ReproBench, and GoldStandardDB).
• Comparison: ARA performs comparably or better than existing agentic systems (ReplicatorAgent, ReproScreener) in terms of agreement with human annotations, despite operating strictly under a document-level setting without access to external repositories or replication packages.
• Disagreement Analysis: Disagreements between ARA and human experts are generally bounded within one ordinal scale level. Disagreement is higher for "Methods" and "Experiments" stages, where human reviewers often rely on implementation details revealed only during full replication, which are not observable from text alone.

Significance and Claims
The paper claims that ARA demonstrates the technical feasibility of using agentic AI systems to approximate human reproducibility judgments at scale. The authors position ARA not as a replacement for human expert peer review, but as a scalable diagnostic complement. By operationalizing reproducibility assessment as a document-level reasoning task, the framework offers a pathway to:

• Accelerate editorial pipelines and large-scale literature screening.
• Provide consistent, transparent, and reproducible quality checks across diverse scientific domains.
• Address the "reproducibility crisis" by identifying reporting gaps before full replication attempts are undertaken.

The authors emphasize that the system's value lies in its ability to function without external artifacts, making it applicable to the vast majority of published literature where code and data are not immediately available. The robustness of the results across different models and the alignment with human judgments on a rigorous benchmark suggest that structured workflow reconstruction can capture meaningful signals of reproducibility beyond simple document parsing.