Directional replicability: when can the factor of two be omitted

Technical Summary: Directional Replicability: When Can the Factor of Two Be Omitted

Problem Statement
The paper addresses the statistical challenge of assessing directional replicability across $n$ independent studies. Unlike standard replicability, which merely requires an effect to be present in at least $r$ studies regardless of sign, directional replicability requires that at least $r$ studies exhibit effects of the same sign (all positive or all negative).

Formally, let $\theta = (\theta_1, \dots, \theta_n)$ be the vector of true effect sizes. The authors define $n_+$ as the count of positive effects and $n_-$ as the count of negative effects. The goal is to test the null hypothesis $H_{r/n}: n_+ < r \land n_- < r$. Rejecting this hypothesis allows one to conclude that there are at least $r$ effects of the same sign ($n_+ \ge r \lor n_- \ge r$).

The standard approach, following Owen (2009), treats $H_{r/n}$ as the intersection of two unilateral hypotheses: $H^+_{r/n}: n_+ < r$ and $H^-_{r/n}: n_- < r$. To control the family-wise error rate, the standard procedure combines the $p$-values for these two unilateral tests and multiplies the minimum by 2 (i.e., $p_{r/n} = 2 \min\{p^+_{r/n}, p^-_{r/n}\}$) to account for the multiple testing of directions. The central question of this work is whether this multiplicative correction factor of 2 is always necessary.

Methodology
The authors assume independent normal estimators $T_i \sim N(\theta_i, 1)$ for the effect sizes. They utilize the Bonferroni partial conjunction method to construct $p$-values for the unilateral hypotheses.

• For the positive direction ($H^+_{r/n}$), the $p$-value is $p^+_{r/n} = (n-r+1)p_{(r)}$, where $p_{(r)}$ is the $r$-th smallest $p$-value from the individual studies.
• For the negative direction ($H^-_{r/n}$), the $p$-value is $p^-_{r/n} = (n-r+1)q_{(r)}$, where $q_{(r)}$ corresponds to the $r$-th smallest $p$-value for the negative direction.

The paper investigates the validity of the uncorrected statistic $p_{r/n} = \min\{p^+_{r/n}, p^-_{r/n}\}$ as a valid $p$-value for the intersection hypothesis $H_{r/n}$ under specific conditions.

Key Contributions and Results

1. Sufficient Condition for Omitting the Factor of Two (Theorem 1):
   The primary theoretical contribution is Theorem 1, which establishes that the factor of 2 is unnecessary when $r$ is sufficiently large relative to $n$. Specifically, if $(n+1)/2 < r \le n$, then $p_{r/n} = \min\{p^+_{r/n}, p^-_{r/n}\}$ is a valid $p$-value for $H_{r/n}$ at significance level $\alpha$.

   • Mechanism: The proof demonstrates that under the condition $r > (n+1)/2$, the events leading to Type I errors for the positive and negative directions are disjoint. Consequently, the probability of rejecting the null hypothesis under the worst-case parameter configuration does not exceed $\alpha$ without the doubling correction.

2. Necessity of Correction for Small $r$ (Section 3):
   The authors provide a counterexample showing that for smaller values of $r$ (specifically $2 \le r \le (n+1)/2$), the uncorrected statistic can lead to Type I error rates exceeding $\alpha$.

   • In this regime, it is possible to have a "discordant" parameter configuration where there are $r-1$ strong positive effects and $r-1$ strong negative effects simultaneously. In such cases, the events leading to rejection in both directions are not disjoint, and the probability of Type I error exceeds the nominal level if the factor of 2 is omitted.
   • Numerical simulations (Figure 1) illustrate that for $n=20$ and $\alpha=0.1$, the Type I error exceeds $\alpha$ for $r \in \{2, \dots, 7\}$ but remains controlled for $r \in \{8, 9, 10\}$, even though the strict condition $r > (n+1)/2$ (i.e., $r > 10.5$) is not met. This suggests the condition in Theorem 1 is sufficient but not necessary.

3. Data-Adaptive Selection of $r$:
   The paper outlines a sequential testing procedure for situations where $r$ is not pre-specified. By testing nested hypotheses $H_{k/n}, H_{(k+1)/n}, \dots, H_{n/n}$ starting from $k = \lceil(n+2)/2\rceil$, one can derive a $(1-\alpha)$ lower confidence bound for the maximum number of effects of the same sign.

4. Extension to Other Combining Functions:
   The authors note that while the proof relies on the Bonferroni method, the result likely extends to other combining functions that assume independence, such as the Šidák correction, the Simes method, and the Fisher combining function, though this requires future verification.

Significance and Claims
The paper claims to refine the state-of-the-art methodology for directional replicability analysis. By identifying the condition $(n+1)/2 < r \le n$, the authors demonstrate that researchers can safely omit the conservative "factor of two" correction in scenarios where the required number of replicating effects ($r$) constitutes a majority of the total studies ($n$). This omission increases statistical power without inflating Type I error rates.

The work also addresses the issue of Type III error (inferring the wrong sign). The authors assert that the procedure in Theorem 1 controls Type III error, allowing for valid post-hoc inference of the sign of the replicated effects (positive if $p^+_{r/n} < p^-_{r/n}$, negative otherwise).

Finally, the paper acknowledges that while it focuses on a single hypothesis, many applications involve multiple features. It suggests that the proposed logic could be integrated with existing filtering approaches (e.g., Wang et al., 2022) but leaves the question of whether the factor of 2 can be removed in the context of multiple testing across features as an open problem for future research.