Human-AI Collaboration for Estimating Scientific Replicability

Technical Summary: Human-AI Collaboration for Estimating Scientific Replicability

Problem Statement
Determining whether published scientific findings can be successfully replicated remains a critical challenge in empirical sciences. Existing approaches for assessing replicability suffer from distinct limitations: human judgment is often susceptible to cognitive biases and limited exposure to the broader literature, while automated machine learning models, though scalable, frequently struggle to capture contextual cues and subtle signals of credibility embedded within scientific narratives. The paper posits that neither human-only nor artificial-only approaches consistently achieve high predictive accuracy, creating a need for a hybrid methodology that leverages the complementary strengths of both.

Methodology
The authors propose and evaluate a hybrid prediction market where algorithmic agents trade alongside human participants to estimate the likelihood of a scientific finding being corroborated by a controlled replication study.

• Data and Training:

  • Training Set: Algorithmic agents were trained on the outcomes of 402 prior replication studies across social and behavioral sciences (including Psychology, Economics, Sociology, and others) sourced from projects like the Reproducibility Project: Psychology and the SCORE program.
  • Features: For each research claim, 41 features were extracted from full texts and metadata, including statistical evidence, author characteristics, publication venue metrics, and funding acknowledgments.
  • Test Set: 30 replication outcomes from the SCORE project were held out for live experiments across six domains: economics, sociology, psychology, marketing/organizational behavior, political science, and education.

• Market Model:

  • Artificial Agents: The system utilizes a geometric artificial prediction market where agents are instantiated in feature space at the locations of training data points. Agents specialize in buying "will replicate" or "will not replicate" contracts based on their proximity to specific training datapoints. They interact using a logarithmic market scoring rule.
  • Hybrid Configuration: In the hybrid setup, trained algorithmic agents trade alongside human participants. Agents were initialized with $500 cash to ensure solvency over the 12-hour market duration, while human participants received $25 per market.
  • Experimental Design: Six live events were conducted involving 97 researchers (PhD students, faculty, and professionals). Markets ran for 12 hours with a one-second trading resolution. The design included three market types for some domains: human-only, artificial-only, and hybrid. Participants were provided with full paper texts and identified claims two days prior to trading.

• Evaluation Metrics:

  • Performance was measured using Mean Absolute Error (MAE) against ground-truth replication outcomes.
  • Hyperparameters (lambda, liquidity, cash allocation) were tuned to balance agent and human influence and ensure realistic market dynamics (e.g., preventing premature convergence or negligible trading).

Key Results
The study compared the predictive accuracy of hybrid markets against human-only and artificial-only baselines across the six disciplines:

• Overall Performance: Hybrid markets consistently matched or outperformed artificial-only markets in most domains.
• Domain-Specific Outcomes:
  • Sociology & Political Science: Hybrid markets achieved the lowest MAE, significantly outperforming both human-only and artificial-only baselines.
  • Economics: The hybrid market (MAE 0.411) outperformed the artificial market (MAE 0.452) but was slightly less accurate than the human-only market (MAE 0.414).
  • Psychology: The human-only market performed best (MAE 0.378), with hybrid and artificial markets showing comparable, slightly higher error rates.
  • Marketing & Education: Artificial markets slightly outperformed hybrid markets in these specific domains (e.g., Marketing MAE: Artificial 0.430 vs. Hybrid 0.490).
• Human Behavior: Post-experiment surveys revealed that human participants primarily relied on epistemic beliefs (personal assessment of replicability) rather than purely profit-maximizing strategies. While some engaged in trend-following, many reported confusion or a lack of structured strategy, suggesting that human contribution is driven more by domain intuition than sophisticated market tactics.

Key Contributions

1. Hybrid Framework: The paper introduces a functional hybrid prediction market architecture that integrates algorithmic pattern recognition (trained on hundreds of prior studies) with real-time human domain expertise.
2. Empirical Validation: Through live experiments, the authors demonstrate that hybrid systems can produce more accurate and reliable replication forecasts than artificial-only systems in most tested contexts, and match or exceed human-only performance in specific domains.
3. Insight into Collaboration: The work provides evidence that human-AI collaboration in forecasting does not merely average out errors but can leverage human contextual reasoning to correct or refine algorithmic predictions, particularly in complex evaluative tasks.

Significance and Claims
The authors claim that this work offers a scalable framework for strengthening evidence assessment in research ecosystems. They argue that thoughtfully designed hybrid human-AI systems can improve reliability in scientific evaluation by combining the speed and breadth of algorithmic agents with the contextual insight of human experts. The paper modestly concludes that while performance varies by domain, the hybrid approach represents a viable and promising direction for scientific forecasting, provided that system design carefully balances the influence of human and algorithmic participants. The authors acknowledge limitations regarding the small scale of the test set and variability in participant engagement, noting that larger-scale deployments are necessary to assess robustness across additional fields.