Evaluating the applicability of replication success metrics in animal-to-human translation: A simulation study

Technical Summary: Evaluating the Applicability of Replication Success Metrics in Animal-to-Human Translation

Problem Statement
Biomedical research faces a significant "translation failure" challenge, where interventions demonstrating promise in preclinical animal studies frequently fail to reproduce efficacy in human clinical trials. Estimates suggest that between two-thirds and 95% of therapeutics safe and effective in animals fail in humans. A primary driver of this failure is the lack of reproducibility and suboptimal study design in animal research, including small sample sizes, lack of randomization/blinding, and publication bias.

While numerous metrics exist to evaluate "replication success" (the consistency of results between an original study and a direct replication in the same population), their applicability to "translation" (assessing consistency between animal and human studies) remains unclear. Translation differs fundamentally from replication because the two studies investigate different populations with different experimental designs, often resulting in different effect sizes. The current standard for assessing translation success is often the "two-trials rule" (requiring statistical significance in both species), but alternative metrics developed for replication have not been rigorously tested in this context.

Methodology
The authors conducted a simulation study to evaluate nine distinct metrics proposed for replication success to determine their performance in an animal-to-human translation setting.

• Data Generation: Synthetic data were generated based on parameters from a real-world meta-analysis by Terstappen et al. regarding prenatal amino acid supplementation and maternal blood pressure. The simulation modeled $k$ animal studies and one human study.
• Simulation Conditions: A fully factorial design created 648 scenarios varying:
  • True effect sizes ($\mu_A, \mu_H$): Null, small (−4.44 mmHg), and large (−24.37 mmHg).
  • Heterogeneity ($\tau^2_A, \tau^2_H$): None, low (19.5), and high (291.1).
  • Animal sample size ($n_A$): Small (10) and larger (20) per group.
  • Number of pooled animal studies ($k$): 2, 3, 4, and 5.
  • Human sample size ($n_H$): Fixed at 107 per group (80% power for small effect).
• Decision Criteria: The study simulated three criteria for proceeding to human trials:
  1. Strict: Human study proceeds only if the pooled animal meta-analysis is statistically significant ($\alpha=0.025$).
  2. Lenient: Human study proceeds if the pooled animal effect is beneficial (negative), regardless of significance.
  3. No Criterion: All simulated pairs are evaluated regardless of animal results.
• Metrics Evaluated:
  1. Significance criterion (Two-trials rule).
  2. Meta-analysis (Fixed-effects).
  3. Replication Bayes Factor (BF).
  4. Unweighted Edgington's method.
  5. Weighted Edgington's method (human study weighted twice as heavily).
  6. Golden sceptical p-value (three versions: no, moderate, and high shrinkage).
  7. Controlled sceptical p-value.
• Performance Measures: The study assessed the proportion of successful translations ($P$) under two main interpretations:
  • False Positive Rate (Type I Error): Under null effects in both species.
  • Translation Power: Under non-null effects in both species.

Key Results
The performance of the metrics varied significantly depending on the simulation conditions, particularly regarding heterogeneity and effect size mismatches.

• False Positive Control: Under the null hypothesis with no heterogeneity, most metrics (except Meta-analysis and Replication BF) controlled the overall Type I error rate at the theoretical $\alpha^2$ (0.000625). However, as heterogeneity increased—particularly in human studies—most metrics became liberal, inflating the false positive rate. The Meta-analysis and Replication BF showed very high Type I error rates (up to 40% in extreme heterogeneity scenarios) because they flag success if either study shows a strong result.
• Translation Power: Power was generally constrained by the weaker evidence of the two findings. Small animal sample sizes ($n_A=10$) resulted in low translation power, even when true effects existed.
  • Meta-analysis: Frequently indicated success if either species found strong evidence, leading to high power but poor specificity (high false positives).
  • Two-trials Rule & Golden Sceptical (No Shrinkage): These were the most conservative, often yielding the lowest power, especially when animal studies were underpowered.
  • Controlled Sceptical P-value & Weighted Edgington: These performed relatively consistently across scenarios, balancing power and error control.
• Impact of Heterogeneity: Increasing heterogeneity generally reduced translation power and increased false positive rates. The Two-trials rule was least affected by changes in human heterogeneity compared to other metrics.
• Impact of Pooling ($k$): Increasing the number of pooled animal studies generally improved power when animal effects were non-null and heterogeneity was low. However, under high animal heterogeneity, increasing $k$ sometimes decreased the overall Type I error rate or had negligible benefits.
• Strict Criterion Effects: Conditioning the human study on a significant animal finding inflated animal effect size estimates. Under this strict criterion, the Two-trials rule and Edgington methods controlled Type I error at level $\alpha$ (0.025) rather than $\alpha^2$.

Key Contributions

1. Systematic Evaluation: This is the first study to systematically evaluate nine replication success metrics specifically within the context of animal-to-human translation, moving beyond the standard two-trials rule.
2. Identification of Metric Limitations: The study demonstrates that metrics designed for direct replication (where studies are exchangeable) often fail in translation settings where studies are conceptually different and effect sizes are expected to shrink or vary. Specifically, the Meta-analysis and Replication BF are shown to be inappropriate for translation success assessment due to their high false positive rates and tendency to ignore evidence mismatches.
3. Context-Dependent Recommendations: The authors provide nuanced recommendations based on study goals:
   • To infer an effect exists in both species with strict error control: Use the Significance criterion, Unweighted Edgington, or Golden Sceptical (no shrinkage).
   • To assess evidence primarily in humans while accounting for animal data: Use Weighted Edgington.
   • For consistent performance across varying conditions: Use the Controlled Sceptical p-value.
4. Emphasis on Study Design: The results highlight that statistical metrics alone cannot compensate for poor study design. Small animal sample sizes and high heterogeneity severely limit the ability of any metric to correctly identify translation success.

Significance and Claims
The paper claims that while replication success metrics can inform translation assessments, no single metric is uniformly optimal. The utility of these metrics is heavily dependent on the underlying evidence (sample sizes, heterogeneity) and the specific assumptions of the researcher (e.g., whether effect shrinkage is expected).

The authors emphasize that these metrics should not be viewed as standalone decision rules for progressing to human trials. Instead, they should be used as quantitative tools within broader frameworks (such as PATH or GALENOS) that consider mechanistic rationale, safety, and pharmacology. The study concludes that a combination of metrics, chosen based on their specific strengths and limitations relative to the available evidence, offers the most robust approach to evaluating the translation of animal findings to human outcomes. The authors explicitly state that their findings are modest regarding specific biomedical contexts, as the simulation parameters were derived from a single meta-analysis, and the study focuses on statistical translation rather than the complex biological mechanisms of translation.