Mixture priors for replication studies

1. Problem Statement

Scientific research faces a "replication crisis," where a significant portion of published findings fail to replicate. A central challenge is quantifying the extent to which a replication study successfully replicates an original result.

• Limitations of Existing Methods:
  • Symmetric Approaches: Meta-analytic pooling often assumes exchangeability between studies, which is inappropriate when explicitly testing the credibility of a specific original finding.
  • Asymmetric Approaches: Methods like Power Priors and Hierarchical Models exist but can suffer from computational complexity (e.g., calculating normalizing constants for Power Priors) or potential issues with improper priors when testing specific hypotheses (e.g., complete discounting of original data).
  • Conflict Handling: There is a need for a method that adaptively "borrows" information from the original study when results align but discounts it when they conflict, without forcing a complete pooling or total dismissal.

2. Methodology

The authors propose a novel Bayesian Mixture Prior framework for analyzing replication data.

Core Concept

The method uses a mixture of two distributions as the prior for the replication analysis:

1. Informative Component: The posterior distribution derived from the original study ($\hat{\theta}_o$).
2. Non-informative Component: A robust, non-informative prior (typically a unit-information normal distribution centered at 0 with large variance).

The mixture prior for the effect size $\theta$ is defined as:
$$\pi(\theta | \hat{\theta}_o, \omega) = \omega \cdot N(\theta | \hat{\theta}_o, \sigma^2_o) + (1 - \omega) \cdot N(\theta | \mu, \tau^2)$$
Where $\omega \in [0, 1]$ is the mixture weight determining the degree of information borrowing.

• $\omega = 1$: Complete pooling (full reliance on original data).
• $\omega = 0$: Complete discounting (reliance solely on replication data).

Two Strategies for the Mixture Weight ($\omega$)

1. Fixed Weight: $\omega$ is set to a predetermined value based on expert knowledge, empirical Bayes estimation, or reverse-Bayes analysis (tipping-point analysis). The resulting posterior remains a mixture of normals with closed-form solutions.
2. Random Weight (Uncertain Weight): $\omega$ is treated as a random variable with a Beta prior distribution, $\omega \sim \text{Beta}(\eta, \nu)$. This allows the data to update the uncertainty regarding the compatibility between studies. The marginal posterior for $\theta$ remains analytically tractable, effectively replacing the fixed $\omega$ with its prior expectation in the likelihood calculations.

Hypothesis Testing via Bayes Factors

The framework derives closed-form Bayes Factors (BF) for two key scientific questions:

1. Consistency Testing: Testing $H_c: \omega = 1$ (studies are consistent) vs. $H_d: \omega = 0$ (studies are inconsistent/original data should be discarded). The BF is the ratio of marginal likelihoods under these point hypotheses.
2. Effect Size Testing: Testing $H_0: \theta = 0$ (no effect) vs. $H_1: \theta \neq 0$ (effect exists), utilizing the mixture prior to incorporate historical data adaptively.

Posterior Predictive Checks (PPC)

The authors utilize PPCs to validate model fit. They demonstrate that the posterior predictive distribution is analytically tractable (a mixture of normals), allowing for the calculation of posterior predictive p-values (PPPs) without simulation.

3. Key Contributions

• Novel Framework: Introduces a specific application of mixture priors to replication studies, distinct from their use in meta-analysis or power priors.
• Analytical Tractability: Unlike normalized power priors which often require numerical integration for normalizing constants, the proposed mixture prior yields closed-form solutions for the marginal likelihood, posterior distributions, and Bayes factors under a normal meta-analytic framework.
• Robustness to Conflict: The method naturally handles discrepancies. If the replication conflicts with the original study, the posterior weight shifts toward the non-informative component, effectively discounting the original data without forcing an improper prior.
• Software Implementation: Development of the open-source R package repmix, which implements the proposed methodology, making it accessible to practitioners.

4. Results (Case Study: "Moral Credentialing")

The authors applied the method to data from the "Many Labs 3" project, specifically the "moral credentialing" experiment (original study + 3 replications).

• Consistent Replications (Rep 1 & 2):
  • The posterior distribution for $\omega$ concentrated near 1.
  • The mixture prior significantly narrowed the 95% Highest Posterior Density Intervals (HPDI) for the effect size compared to using replication data alone (reductions of ~42-45%).
  • Bayes factors provided strong evidence for the presence of an effect and consistency with the original study.
• Conflicting Replication (Rep 3):
  • The replication effect was negative ($\hat{\theta}_{r3} = -0.18$), contradicting the original positive effect.
  • The posterior for $\omega$ concentrated near 0, indicating the model successfully discounted the original data.
  • The resulting HPDI for the effect size was wider than the replication-only analysis, correctly reflecting the uncertainty and conflict.
  • Bayes factors provided strong evidence for the null hypothesis (no effect) and inconsistency with the original study.
• Pooled Replications:
  • When pooling all three replications, the combined evidence aligned with the original study, yielding a strong effect estimate and a narrow HPDI.
• Comparison with Other Methods:
  • vs. Power Priors: The mixture prior produced sharper posteriors when data agreed and wider ones when they disagreed. It avoided the computational burden of normalizing constants required by normalized power priors.
  • vs. Hierarchical Models: While hierarchical models also detected conflict (by inflating heterogeneity $\tau$), they resulted in posteriors that were overly diffuse (shifting toward zero). The mixture prior provided a more intuitive and direct measure of discounting via $\omega$.

5. Significance and Conclusion

• Intuitive Interpretation: The mixture weight $\omega$ offers a more direct and interpretable measure of "discounting" historical data compared to the power parameter ($\alpha$) in power priors or the heterogeneity variance ($\tau^2$) in hierarchical models.
• Methodological Rigor: The approach resolves the issue of improper priors when testing for complete inconsistency ($\omega=0$), a known difficulty in power prior formulations.
• Flexibility: The framework is adaptable to various effect sizes and data types (assuming normality) and can be extended to include multiple original studies.
• Practical Utility: By providing the repmix R package, the authors facilitate the adoption of rigorous, Bayesian replication analysis, moving the field toward more transparent and quantitative assessments of scientific credibility.

The paper concludes that mixture priors represent a valuable, intuitive, and computationally efficient alternative for evidence synthesis in replication studies, particularly when the goal is to assess the credibility of specific historical findings rather than just estimating a global average effect.