Technical Summary: Testing Imprecise Hypotheses

1. Problem Statement

The paper addresses the fundamental challenge of testing goodness-of-fit hypotheses when the null hypothesis is not a single, precise distribution but rather a composite set representing "imprecise" knowledge. In many scientific applications, such as high-energy physics, the null distribution $P_0$ is known only up to systematic uncertainties or approximation errors (e.g., via simulators).

Standard goodness-of-fit testing typically evaluates $H_0: P = P_0$ versus $H_1: P \neq P_0$. The authors instead adopt the tolerant testing framework, which formulates the problem as distinguishing between two composite hypotheses:
$$H_0: d(P, P_0) \leq \epsilon_0 \quad \text{versus} \quad H_1: d(P, P_0) \geq \epsilon_1$$
where $d$ is a divergence metric (e.g., $\ell_p$ norm, total variation), $\epsilon_0 > 0$ is a tolerance (imprecision) parameter defining a "systematic uncertainty ball" around $P_0$, and $\epsilon_1 > \epsilon_0$ is the separation parameter.

The primary goal is to characterize the critical separation (or critical separation radius), defined as the smallest gap $\epsilon_1 - \epsilon_0$ required to distinguish the hypotheses with non-trivial power uniformly over the null and alternative classes. The paper investigates the information-theoretic trade-off between the size of the tolerance $\epsilon_0$ and the statistical power of the test across several canonical models.

2. Methodology

The authors employ the minimax framework for hypothesis testing, rooted in the works of Ingster, Lepski, and others. The methodology involves:

1. Model Selection: The analysis is conducted across three primary settings:

   • The Gaussian sequence model ($X \sim N(v, \sigma^2 I_d)$) under $\ell_p$ metrics.
   • The smooth Gaussian white noise model ($dX(t) = f(t)dt + \sigma dW(t)$) under $L_p$ metrics.
   • The smooth density model (estimating an unknown density $f$) under $L_p$ metrics.

2. Upper Bounds (Test Construction):

   • For non-smooth norms (e.g., $\ell_1$), the authors analyze plug-in tests based on the empirical norm (e.g., $T_1(X) = \|X\|_1 - E_{P_0}\|X\|_1$).
   • For smooth norms (even integers $p$), they utilize debiased plug-in statistics that subtract bias terms derived from Hermite polynomials to achieve optimal rates.
   • They demonstrate that for large tolerance, estimation-based tests (rejecting if an estimator of the divergence exceeds a threshold) become optimal.

3. Lower Bounds (Impossibility Results):

   • The authors construct moment-matching distributions (mixtures of Gaussians) to prove minimax lower bounds.
   • They utilize the duality between moment-matching and best polynomial approximation (a technique from Ingster, Wu, Yang, and others) to determine the maximum functional separation achievable by distributions that share a certain number of moments.
   • These constructions show that no test can distinguish hypotheses closer than the derived critical separation.

3. Key Contributions and Results

A. The Gaussian Sequence Model

The paper provides a sharp characterization of the critical separation for testing $\|v\|_p \leq \epsilon_0$ vs. $\|v\|_p \geq \epsilon_1$.

1. Non-Smooth Norms ($\ell_1$):
The authors identify three distinct regimes for the critical separation $\epsilon_1 - \epsilon_0$ as a function of the tolerance $\lambda = \epsilon_0/\sigma$:

• Free Tolerance Regime ($0 \leq \lambda \lesssim \sqrt{d}$): The critical separation remains constant at the standard goodness-of-fit rate $d^{3/4}/\sqrt{n}$. The test power does not degrade as $\epsilon_0$ increases within this range.
• Interpolation Regime ($\sqrt{d} \lesssim \lambda \lesssim d$): The critical separation scales as $d^{1/2} \lambda^{1/2}$. This regime is unique to tolerant testing, interpolating between the goodness-of-fit rate and the functional estimation rate.
• Functional Estimation Regime ($\lambda \gtrsim d$): The critical separation coincides with the tolerance $\epsilon_0$ (scaling as $d$), matching the minimax rate for estimating the $\ell_1$ norm.

Key Finding on Sub-optimality: The paper establishes that the classical $\chi^2$-test (Pearson's test), while minimax optimal for simple nulls ($\epsilon_0=0$), is sub-optimal for tolerant testing. It possesses a limited tolerance of only $d^{1/4}/\sqrt{n}$ before losing power. In contrast, a simple plug-in test ($T = \sum |X_j|$) achieves the optimal tolerance of $d^{1/2}/\sqrt{n}$.

2. Smooth Norms ($\ell_p$ for even $p$):
For smooth functionals, the authors show that the interpolation regime disappears. The critical separation remains constant at the goodness-of-fit rate ($d^{1/2p}/\sqrt{n}$) until $\epsilon_0$ reaches the functional estimation rate, after which it transitions directly to the estimation rate. This confirms that for smooth norms, tolerant testing and functional estimation are tightly linked across the entire spectrum.

3. Non-Smooth Norms ($\ell_p$ for non-even $p$):
For non-even $p$ (including odd integers $>2$), the behavior resembles the $\ell_1$ case, exhibiting an interpolation regime, though the specific rates depend on the parity of $p$ and the bias structure of the plug-in statistic.

B. Extensions to Infinite-Dimensional Models

Using discretization arguments and equivalence results (reducing white noise and density models to Gaussian sequence or Multinomial models), the authors extend their findings:

• Smooth Gaussian White Noise: The critical separation under $L_p$ norms interpolates between simple null testing rates and functional estimation rates, dependent on the smoothness parameter $s$.
• Smooth Density Models: Similar equivalence holds for density testing under $L_1$ norms, showing that tolerant testing in density models is as hard as in Gaussian white noise models, without requiring a lower bound on the density (a standard requirement for Le Cam deficiency results).

4. Significance and Claims

The paper claims to provide a systematic characterization of the information-theoretic trade-off between the size of the null hypothesis (tolerance) and the power of the test.

• Robustness Quantification: The existence of a "free tolerance regime" allows practitioners to quantify the robustness of goodness-of-fit tests. It identifies the maximum systematic uncertainty a test can tolerate without sacrificing its ability to detect deviations of a specific magnitude.
• Sub-optimality of Classical Tests: A significant contribution is the demonstration that widely used tests (like the $\chi^2$-test) are not robust to misspecification in the tolerant setting, whereas simple alternatives (like the $\ell_1$ plug-in test) are.
• Unified Framework: The work unifies the study of goodness-of-fit testing and functional estimation, showing that tolerant testing naturally interpolates between these two problems.
• Application to High-Energy Physics: The authors argue that the tolerant testing framework provides a rigorous basis for handling systematic uncertainties in fields like high-energy physics, where null distributions are approximated via simulators. They suggest that inverting tolerant tests can yield "tolerance factors" to assess the sensitivity of p-values to the accuracy of the null model.

The authors remain modest regarding their results, noting that while they provide sharp bounds for specific canonical models, the interpolation regime does not always exist (e.g., for smooth norms), and the results for general non-smooth norms are conjectured to be tight up to polylogarithmic factors. They do not propose new experimental protocols but rather offer theoretical tools to interpret and design tests in the presence of imprecise null hypotheses.