Sensitivity Analysis for False Discovery Rate Estimation with Published p-Values

1. Problem Statement

The scientific community faces a significant challenge in estimating the False Discovery Rate (FDR) using only published p-values. Published literature is subject to publication bias (or selection bias), where studies with statistically significant results (typically $p < 0.05$) are more likely to be published than those with non-significant results.

• The Core Issue: Existing FDR estimators (e.g., Jager & Leek, 2014; Johnson et al., 2017) rely on specific assumptions about the Selection Probability Model (SPM)—the mechanism determining which p-values get published.
• The Gap: There is a lack of formal research quantifying how misspecification of this selection model impacts the bias and variance of FDR estimators. If the assumed selection rule (e.g., "publish if $p < 0.05$") does not match the true, often unknown, selection mechanism, the resulting FDR estimates may be misleading (either overly conservative or liberal).
• Goal: The authors aim to provide a framework to assess the robustness of FDR estimates by deriving general expressions for bias and variance under arbitrary selection models, enabling a sensitivity analysis.

2. Methodology

A. Theoretical Framework

The authors establish a rigorous mathematical framework combining Selective Inference and Mixture Models:

1. P-value Mixture Model: They assume p-values $\{P_i\}$ follow a mixture distribution where a proportion $\pi_0$ are true nulls (Uniform[0,1]) and $(1-\pi_0)$ are false nulls (stochastically smaller, e.g., Beta distribution).
2. Selection Probability Model (SPM): They define a non-increasing function $\mu(p)$ representing the probability of a p-value being selected (published) given its value. This generalizes standard rules like "publish if $p \leq 0.05$."
3. Post-Selection FDR: They define the post-selection FDR as the expected proportion of false discoveries among selected discoveries.

B. The Estimator

The paper focuses on a specific FDR estimator (motivated by Hung & Fithian, 2020) which:

1. Adjusts observed p-values: $p'_i = p_i / \alpha$ (where $\alpha$ is the significance threshold).
2. Applies a method-of-moments estimator for the null proportion ($\pi_0$) using a tuning parameter $\lambda$:
   $$\hat{\pi}_0 = \frac{1}{n'} \sum_{i=1}^{n'} \frac{\mathbb{I}(\lambda, 1](p'_i)}{1 - \lambda}$$
   The FDR estimate is then derived from this $\hat{\pi}_0$.

C. Derivation of Bias and Variance

The core methodological contribution is the derivation of closed-form expressions for the bias and variance of this estimator under arbitrary SPMs ($\mu(p)$), not just the assumed one.

• Bias Expression: Derived as a function of the true selection model, the assumed model, the tuning parameter $\lambda$, the null proportion $\pi_0$, and the power of the tests.
• Variance Expression: An asymptotic approximation based on the multivariate normal approximation of the multinomial distribution.

3. Key Contributions

1. General Bias and Variance Formulas: The paper provides the first general, closed-form expressions for the bias and variance of a standard FDR estimator when the selection model is misspecified.
   • Key Insight: The bias is not necessarily positive (as is often assumed in standard settings) nor monotone in $\lambda$. It can be negative (leading to underestimation of FDR) if the true selection mechanism favors very small p-values more strongly than the assumed model (e.g., if $p < 0.025$ is much more likely to be published than $0.025 < p < 0.05$).
2. Sensitivity Analysis Framework: The authors demonstrate how to use these expressions to perform a sensitivity analysis. Instead of providing a single point estimate, researchers can report FDR estimates across a range of plausible selection models (parameterized by $\rho$ or $\eta$).
3. Validation with Replication Data: The theoretical bias expressions were validated against the "gold standard" FDR estimates from the Open Science Collaboration (2015) replication project (Johnson et al., 2017), where both original and replication p-values were available. The bias-corrected estimates using only original p-values closely matched the gold standard.

4. Results

Theoretical Findings

• Bias Direction: The bias depends heavily on the relationship between the assumed and true selection models.
  • If the assumed model is correct, the bias is positive and decreases as power increases.
  • If the assumed model is incorrect (e.g., assuming a flat selection probability for $p < 0.05$ when the true probability drops sharply after $0.025$), the bias can become negative, causing the FDR to be severely underestimated.
• Impact of Parameters:
  • High prior null probability ($\pi_0$) and low statistical power exacerbate the risk of negative bias under misspecification.
  • The tuning parameter $\lambda$ significantly affects the bias magnitude and direction.

Empirical Applications

1. Open Science Collaboration (Psychology):
   • Using the bias expressions, the authors adjusted the FDR estimates derived from the 73 original p-values.
   • The bias-corrected estimates (ranging from 0.37 to 0.65 depending on $\lambda$) aligned closely with the Johnson et al. (2017) gold standard estimates (approx. 0.49), validating the theoretical bias formulas.
2. Medical Journal Abstracts (Jager & Leek, 2014):
   • The authors applied the sensitivity analysis to p-values from the Lancet.
   • They varied the selection model parameter $\rho$ (representing the relative likelihood of publishing $p < 0.025$ vs. $0.025 < p < 0.05$).
   • Result: The estimated FDR varied significantly based on the assumed selection mechanism.
     • Under the standard "publish if $p < 0.05$" assumption ($\rho=1$), the bias-corrected FDR was $\approx 0.19$.
     • Under a stricter selection assumption ($\rho=0.4$, meaning small p-values are 2.5x more likely to be published), the bias became negative, and the corrected FDR rose to $\approx 0.30$.

5. Significance and Implications

• Policy and Replicability: The paper argues that policymakers and researchers should not rely on a single FDR estimate derived from a potentially arbitrary selection assumption. Instead, they should report a range of estimates corresponding to a spectrum of plausible selection models.
• Diagnosing Scientific Health: The provided expressions allow for a "health check" of the scientific ecosystem. If FDR estimates are highly sensitive to the selection model, it indicates that the current publication landscape is heavily biased, and conclusions about replicability are unstable.
• Practical Utility: The method allows for FDR estimation using only published p-values (which are readily available) while explicitly accounting for the uncertainty of the selection mechanism, bridging the gap between theoretical selective inference and practical meta-science.

In summary, this paper moves the field beyond "blind" FDR estimation by providing the mathematical tools to quantify and correct for the uncertainty introduced by publication bias, offering a more robust approach to evaluating the reliability of scientific literature.