Automated reproducibility assessments in the social and behavioral sciences using large language models

Technical Summary: Automated Reproducibility Assessments in the Social and Behavioral Sciences Using Large Language Models

Problem Statement
Reproducibility in the social and behavioral sciences is traditionally evaluated by independent researchers who manually reanalyze original data to verify if published findings can be recovered. While essential, this process is resource-intensive, difficult to scale, and often limited to small-scale efforts (e.g., the Multi100 collaboration involving 507 analysts over several years). The manual reconstruction of empirical workflows—requiring data inspection, variable identification, code translation, and execution—creates a bottleneck for systematic auditing of the literature. Although Large Language Models (LLMs) have demonstrated strong coding capabilities, it remains unknown whether they can reliably perform end-to-end computational reproducibility assessments, interpret analytical decisions from study materials, and avoid errors or unsupported assumptions.

Methodology
The authors developed an agentic LLM pipeline to automate reproducibility assessments across a corpus of $N=180$ published empirical studies from the Systematizing Confidence in Open Research and Evidence (SCORE) project, covering psychology, political science, and economics.

• Data and Inputs: The pipeline utilized pre-defined empirical claims, original study datasets (or replication data where original data were unavailable), and article text. The text was converted from PDF to Markdown using Mistral OCR to preserve layout, equations, and tables.
• Agentic Workflow: The LLM agent was tasked with independently writing and executing statistical code in an isolated sandbox to estimate the effect size tied to the focal claim. The agent was explicitly instructed not to copy reported statistics from the paper but to compute them from the provided data.
• Experimental Design:
  • Runs: Each study underwent five independent runs to assess variability.
  • Models: The primary analysis used Claude Opus 4.7. Robustness was tested using GPT-5.5 and the open-weight GLM-5.1.
  • Information Contexts: To test sensitivity to methodological detail, the pipeline was run under three conditions: (1) full paper text, (2) full paper with the methods section removed, and (3) abstract only.
  • Prompt Framing: Sensitivity analyses compared neutral prompts against "confirmatory" (supportive) and "critical" (skeptical) framings.
• Evaluation Metrics:
  • Effect Size Recovery: The LLM-generated mean effect size ($\bar{d}_{LLM}$) was compared to the original published effect size ($d_{original}$) using Cohen's $d$. Recovery was defined as falling within strict ($\pm 0.05$) and broad ($\pm 0.20$) tolerance regions.
  • Qualitative Conclusion: The LLM's conclusion (support/opposite/inconclusive) was compared to the original study's conclusion.
  • Human Benchmark: For a subset of $N=84$ studies, LLM results were compared against a large-scale human reanalysis effort [2].
• Preprocessing: Studies were excluded if the LLM failed to produce a valid Cohen's $d$ in any of the five runs (11 studies excluded, leaving $N=169$). Outliers with implausibly large effect sizes (e.g., due to conversion errors) were manually reviewed and filtered.

Key Results

• Overall Reproducibility (RQ1): Across the 169 valid studies, the LLM pipeline reached the same qualitative conclusion as the original study in 80% of cases. It recovered the original effect size within the strict $\pm 0.05$ tolerance in 24% of studies and within the broad $\pm 0.20$ tolerance in 50% of studies. Reproducibility was higher when original source data were available compared to when only replication data were used.
• Comparison with Human Reanalysts (RQ2): In the subset with human benchmarks ($N=82$), the LLM matched the original qualitative conclusion in 95% of cases (vs. 83% for humans). The LLM recovered the original effect size within the strict tolerance in 40% of cases, broadly similar to human reanalysts (28%). However, the LLM's effect sizes correlated more strongly with the original published findings ($r=0.46$) than with human reanalysis findings ($r=0.11$), suggesting the LLM followed a narrower, paper-anchored analytical path rather than exploring the full space of defensible specifications that humans might pursue.
• Information Context Sensitivity (RQ3): Reproducibility rates did not significantly differ across information contexts. Whether the model received the full paper, the paper without methods, or only the abstract, the strict tolerance recovery rate remained stable (24%, 24%, and 22%, respectively).
• Robustness: Results were consistent across the three tested models (Claude Opus 4.7, GPT-5.5, GLM-5.1) and were not significantly affected by prompt framing (confirmatory vs. critical).

Significance and Claims
The paper claims that LLMs can serve as a scalable screening tool for reproducibility assessments but explicitly states they are not yet a substitute for expert judgment.

• Scalability: The findings demonstrate that LLMs can automate the computationally intensive task of reanalysis, potentially enabling systematic audits of empirical results that are currently too costly to perform manually.
• Role in Scientific Workflow: The authors position LLMs as a "first-pass" check for journals and meta-scientists to identify whether reported findings can be recovered from available data. This could integrate into submission or post-publication quality control processes.
• Limitations and Caveats: The authors emphasize that LLM-based reanalysis is an emerging capability. They note that some runs yield no viable results and that the models may not capture the full range of defensible analytical choices (the "multiverse" of analysis) that human analysts explore. Furthermore, the study warns against "Goodhart-style" incentives where authors might optimize for passing automated screens rather than ensuring robust science, and cautions that automated checks do not replace the need for replication with newly collected data.
• Generalizability: The use of an open-weight model (GLM-5.1) alongside proprietary models suggests that automated reanalysis is not limited to closed systems.

In summary, the study provides empirical evidence that frontier LLMs can successfully reproduce a substantial share of published findings in the social sciences, offering a promising, scalable instrument to improve the rigor of empirical research while acknowledging current limitations in reliability and the necessity of human oversight.