When Is a Relevance Threshold Statistically Resolvable? Minimax Limits for Effect Classification

Technical Summary: "When Is a Relevance Threshold Statistically Resolvable?"

Problem Statement
The paper addresses the fundamental disconnect between statistical precision and scientific relevance. In regular statistical models, sampling uncertainty contracts at a rate of $n^{-1/2}$, allowing increasingly small deviations from a point null to be detected as sample size ($n$) grows. However, scientific importance is often defined by a threshold $\Delta_n$ (the smallest effect size of interest) below which an effect is considered negligible. This creates a "relative-rate problem": as information accumulates, the scientific threshold $\Delta_n$ may shrink, remain fixed, or vary at a different rate than the statistical error scale $s_n \sim n^{-1/2}$. The central question is: under what conditions can a statistical procedure consistently distinguish between scientifically negligible effects ($|\theta| \le \Delta_n$) and scientifically meaningful effects ($|\theta| > \Delta_n$)?

Methodology and Framework
The author employs a minimax decision-theoretic framework combined with local asymptotic theory.

1. Relevance-Resolution Index: The core quantity is defined as $\lambda_n = \sqrt{n I_0} \Delta_n$, where $I_0$ is the Fisher information at $\theta=0$. This index measures the scientific threshold in units of statistical error.
2. Separation of Classes: The problem is formulated as distinguishing between two separated parameter classes:
   • Negligible: $N_n(\varepsilon) = \{ \theta : |\theta| \le (1-\varepsilon)\Delta_n \}$
   • Meaningful: $M_n(\varepsilon) = \{ \theta : |\theta| \ge (1+\varepsilon)\Delta_n \}$
     where $\varepsilon \in (0,1)$ provides a buffer to avoid the exact boundary.
3. Risk Metric: The performance of a classifier $\phi_n$ is measured by the worst-case sum of Type I and Type II errors: $R_n(\phi_n) = \sup_{\theta \in N_n} E_\theta[\phi_n] + \sup_{\theta \in M_n} E_\theta[1-\phi_n]$. The goal is to characterize the minimax risk $R^*_n$.
4. Asymptotic Regimes: The analysis categorizes behavior based on the limit of $\lambda_n$ as $n \to \infty$:
   • Subcritical: $\lambda_n \to 0$
   • Critical: $\lambda_n \to \kappa \in (0, \infty)$
   • Supercritical: $\lambda_n \to \infty$

Key Contributions and Results

1. Minimax Lower Bounds and Phase Transitions
The paper establishes a minimax lower bound for the classification risk. As $\lambda_n \to \kappa$, the limiting risk satisfies:
$$\liminf_{n \to \infty} R^*_n \ge 2\{1 - \Phi(\varepsilon \kappa)\}$$

• Subcritical Regime ($\kappa = 0$): The lower bound approaches 1. The experiments for negligible and meaningful effects merge (total variation distance $\to 0$). No procedure can perform better than a trivial constant decision rule; consistent classification is information-theoretically impossible.
• Critical Regime ($\kappa \in (0, \infty)$): The problem converges to a non-degenerate Gaussian-shift experiment. The paper derives the exact minimax rule for this limiting composite problem.
  • The optimal cutoff $c^*$ satisfies a specific transcendental equation involving hyperbolic cosines.
  • As $\kappa \downarrow 0$, the optimal cutoff converges to 1 statistical-error unit ($c^* \to 1$), not to the collapsing scientific boundary.
  • The optimal risk improves over the trivial risk at order $\kappa^2$, whereas a simple rule that thresholds exactly at the scientific boundary ($|\hat{\theta}| > \Delta_n$) only improves at order $\kappa^3$.
  • As $\kappa \to \infty$, the simple scientific-boundary rule becomes asymptotically minimax.
• Supercritical Regime ($\kappa \to \infty$): Consistent classification is achievable if a "Uniform Resolution Condition" holds (i.e., the estimator's error is uniformly tight relative to the scale). For regular estimators where $\Delta_n = d n^{-\gamma}$, the critical rate is $\gamma = 1/2$. If $\gamma > 1/2$, resolution is impossible; if $\gamma < 1/2$, it is consistently resolvable.

2. Asymmetric and Moving Regions
The paper demonstrates that the scalar index $\lambda_n$ is insufficient for asymmetric or moving relevance regions $R_n = [-\Delta_{L,n}, \Delta_{U,n}]$. Resolvability is governed by the standardized distances to the lower and upper boundaries ($d_{L,n}$ and $d_{U,n}$) rather than the total width alone. If boundaries move at different rates, one side may remain unresolved even if the total width suggests high resolution.

3. Significance Saturation in Heterogeneous Populations
The paper analyzes the behavior of point-null tests across a population of heterogeneous true effects $\Theta \sim G$.

• Saturation Limit: As information increases, the proportion of statistically significant findings that are scientifically negligible converges to a value determined by the population mass of nonzero effects within the relevance region, rather than the significance level $\alpha$.
• Non-monotone Selectivity: The paper identifies a "significance saturation" phenomenon where point-null significance is not monotonically increasing in its ability to select for relevant effects. At very low information, significance is driven by noise; at very high information, it detects nearly all nonzero effects (including negligible ones). Consequently, there can be an intermediate information level where point-null significance is most selective for scientifically relevant effects (minimizing the "irrelevant discovery share").

Significance and Claims
The paper explicitly states that it does not propose a new equivalence test or a new method for calculating p-values. Instead, its significance lies in:

1. Characterizing Resolvability: It defines the precise information-theoretic limits of when a scientific relevance threshold can be statistically distinguished from zero.
2. Unifying Scales: It links effect size, statistical power, local asymptotic theory, and minimax decision theory through a common information scale ($\lambda_n$).
3. Clarifying Large-Sample Behavior: It resolves the apparent paradox that increasing sample size improves estimation but does not necessarily improve the ability to distinguish "substantive" from "negligible" if the threshold shrinks too fast.
4. Optimal Decision Rules: It provides the exact minimax classifier for the critical regime, showing that optimal decision boundaries do not track the scientific threshold linearly near the unresolved edge but stabilize at one standard error unit.

The framework emphasizes that statistical evidence is a distance measured in uncertainty units, while scientific relevance is a distance measured from a substantive boundary. A valid inferential analysis must maintain visibility of both scales.