Attempts to Reproduce Health Sciences Research Are Uncommon and May Be Declining Over Time: A Systematic Review and Meta-Analysis

Technical Summary: "Attempts to Reproduce Health Sciences Research Are Uncommon and May Be Declining Over Time"

Problem Statement
Despite widespread concerns regarding irreproducible research in health sciences—cited as a driver of wasted resources, patient harm, and eroded public trust—there is a lack of systematic quantification regarding how frequently the scientific community attempts to reproduce prior work. While it is estimated that a significant portion of health science methods and results cannot be reproduced, it remains unclear how commonly dedicated reproducibility studies are conducted to monitor this issue. Furthermore, the temporal trends of these efforts in response to growing reproducibility concerns are not well characterized. This study addresses the knowledge gap regarding the prevalence of reproducibility attempts, the methods used to identify them, and their evolution over time.

Methodology
This systematic review and meta-analysis adhered to PRISMA guidelines and the Cochrane handbook, with a protocol registered a priori on PROSPERO.

• Data Sources: The authors searched MEDLINE, Embase, Web of Science, the Cochrane Controlled Trials Registry, the Cochrane Database of Systematic Reviews, and grey literature from January 1, 2000, to May 31, 2023.
• Inclusion Criteria: English-language studies published after 1999 that estimated the proportion of published research articles dedicated to reproducing prior or concurrent research within health sciences.
• Classification of Reproducibility: Attempts were categorized into three types:
  1. Methodological: Exactly repeating another study's methods.
  2. Results: Comparing results using the same or similar methods (further classified as approximate, re-test, or unspecified).
  3. Inferential: Comparing conclusions, including re-analyses of original data.
• Identification Methods: The review analyzed how studies identified reproducibility attempts: (1) inference based on study language, (2) inference based on the comparability of design elements between studies, or (3) explicit reporting that a study was a reproducibility attempt.
• Statistical Analysis: Prevalence estimates were meta-analyzed using random-effects models (mixed-effects logistic regression) weighted by sample size. Heterogeneity was assessed using Cochrane's Q. Meta-regression with time-varying covariates was employed to evaluate temporal trends, with a statistical significance threshold of p < 0.01. Sensitivity analyses were conducted using earliest, median, and latest publication dates of underlying data to account for multi-year datasets.

Key Results

• Study Selection: From 55,218 screened articles, 41 studies were included, representing 30 distinct Clarivate research areas across biomedical/preclinical (n=10), clinical (n=26), population health (n=2), and multiple domains (n=3).
• Overall Prevalence: The pooled estimated prevalence of reproducibility attempts in health sciences literature between 2000 and 2023 was 1.6% (95% CI: 0.6–4.2%).
• Variability: Prevalence varied significantly by research domain, research area, and the method used to identify attempts.
  • By Domain: Population health showed the highest prevalence (3.5%), followed by biomedical/preclinical (3.0%), clinical (1.3%), and multiple domains (0.1%).
  • By Identification Method: Studies inferred via language (12.6%) or design comparability (11.9%) showed much higher prevalence than those explicitly reported as reproducibility attempts (0.3%).
  • By Research Area: Estimates ranged from 0.0% to 77.8%, though 22 of the 30 research areas were based on single studies.
• Temporal Trends: Meta-regression identified a non-significant increase in the odds of reproducibility attempts per year between 2000 and 2015 (49.8% increase, p=0.03). However, this was followed by a significant decrease between 2016 and 2023, with a 50.2% decrease in odds per year (p < 0.001).
• Sensitivity: The finding of a post-2016 decline was sensitive to analytic assumptions. When sensitivity analyses assigned different publication years (earliest, median, latest) to multi-year data, the observed declines became non-significant, suggesting the trend should be interpreted with caution.

Significance and Claims
The authors conclude that attempts to reproduce health sciences research constitute a small proportion of the literature and may be declining. The study highlights several critical implications:

1. Need for Increased Effort: Despite the recognized importance of reproducibility, the proportion of literature dedicated to it is low and appears to have decreased since 2016. The authors argue for greater investment in reproducibility attempts to identify irreproducible science and build robust evidence bases.
2. Standardization of Classification: The significant variation in prevalence estimates based on how attempts are identified (explicit vs. inferred) underscores a need for standardized methods to classify and report reproducibility attempts. Distinguishing between deliberate and incidental reproducibility attempts is necessary for accurate meta-science.
3. Monitoring Specific Domains: The authors note a lack of estimates for methodological and inferential reproducibility specifically. They suggest that monitoring these types is crucial, particularly given existing interventions (e.g., protocol registries, reporting guidelines) aimed at improving methodological rigor.
4. Future Research Directions: The paper calls for primary studies to quantify the prevalence of different reproducibility types and to investigate the relationship between the prevalence of reproduction attempts and other metrics of scientific robustness (e.g., risk of bias, retraction rates).

The study emphasizes that research reproducibility is not being routinely or consistently monitored. Consequently, there is an urgent need to increase the volume of reproducibility attempts and to standardize their classification and reporting to better evaluate the trustworthiness of scientific findings.