Technical Summary: On Focusing Statistical Power for Searches and Measurements in Particle Physics

1. Problem Statement

Particle physics experiments rely heavily on the Generalized Likelihood Ratio Test (GLRT) for both searching for new phenomena and measuring known parameters. These analyses typically involve composite hypothesis tests (e.g., testing a null hypothesis $H_0: \mu = \mu_0$ against a composite alternative $H_1: \mu \neq \mu_0$).

While the Neyman-Pearson lemma guarantees that the Likelihood Ratio Statistic (LRS) is optimal for simple-vs-simple hypothesis tests, it does not guarantee optimality for composite tests. In general settings, no single test statistic is simultaneously most powerful across all possible values of the alternative parameter space. Consequently, the choice of the LRS implicitly dictates how statistical power is distributed across the parameter space, often resulting in suboptimal sensitivity in specific regions of interest. Furthermore, constructing valid confidence intervals for these complex models often requires computationally expensive Monte Carlo (MC) pseudo-experiments, particularly when asymptotic approximations (like Wilks' theorem) fail due to small sample sizes or boundary effects.

2. Methodology

The authors propose a Focused Test Statistic (FTS) as a drop-in replacement for the standard LRS. The FTS is defined as:

$$T_f(D; \mu_0) = -2 \log \left( \frac{p(D; \mu_0)}{\int_{\Theta} p(D; \mu) f(\mu) d\mu} \right)$$

Where:

• $p(D; \mu)$ is the likelihood of the data given parameter $\mu$.
• $f(\mu)$ is a focus function (a weight function) defined by the analyst, not the data. This function represents physics-motivated regions of interest (e.g., a Gaussian centered on a Standard Model prediction or a null hypothesis).
• The denominator represents a weighted average of the likelihood over the alternative hypothesis space.

Key Methodological Features:

• Frequentist Validity: Although the FTS has Bayesian roots (resembling a Bayes factor), the authors employ it strictly within a frequentist framework. The resulting confidence intervals are constructed by inverting the hypothesis tests.
• Optimality: The FTS is proven to minimize the expected length of the confidence interval with respect to the marginal distribution induced by the focus function $f(\mu)$, while maintaining exact coverage ($1-\alpha$) for every $\mu \in \Theta$.
• Bias vs. Power: The method intentionally introduces a form of technical bias (reduced power in regions where $f(\mu)$ is small) to maximize power in regions where $f(\mu)$ is large. This is distinct from the CLs method; FTS guarantees correct coverage across the entire parameter space.
• Efficient Calibration: To avoid the computational cost of traditional MC methods for determining critical values $C_{\mu_0}$, the authors utilize Quantile Regression (QR) via machine learning. This allows for the rapid estimation of critical values as a function of $\mu_0$, enabling the construction of valid confidence intervals even in small-sample regimes where asymptotics fail.

3. Key Contributions

1. Introduction of FTS: A new test statistic that allows analysts to "focus" statistical power on specific, physics-motivated regions of the parameter space without sacrificing the validity of confidence intervals.
2. Theoretical Guarantees: Proof that FTS intervals are the shortest among all valid intervals when averaged over the distribution induced by the focus function, while maintaining exact frequentist coverage.
3. Computational Framework: A demonstration of using machine learning (quantile regression) to efficiently calibrate critical values for both LRS and FTS, reducing the need for massive Monte Carlo simulations.
4. Empirical Validation: Application of the method to two distinct high-energy physics scenarios: a Higgs boson coupling measurement and a Dark Matter (WIMP) search.

4. Results

The authors evaluated the FTS against the standard LRS using simulated datasets from the ATLAS experiment (Higgs $\to \tau\tau$) and a dataset inspired by the LZ dark matter experiment.

Higgs Coupling Measurement

• Setup: Measuring the signal strength $\mu$ of Higgs to tau coupling, with the Standard Model prediction at $\mu=1$.
• Performance:
  • Using a wide focus ($s=2.4$) centered at $\mu=1$, the FTS produced 68% confidence intervals 13% shorter than LRS when the true signal was $\mu=1$.
  • Using a narrow focus ($s=1.2$), the intervals were 21% shorter than LRS at $\mu=1$.
  • The improvement remained robust even when the true signal strength deviated modestly from the focus center.
• Observables: Gains were observed using both the reconstructed Higgs mass and the visible mass observables.

Dark Matter (WIMP) Search

• Setup: Searching for WIMPs where the null hypothesis is $\mu=0$ (no signal). The focus was centered at $\mu=0$.
• Performance:
  • When no signal exists ($\mu^*=0$), the FTS with a wide focus produced 68% confidence intervals 47% shorter than LRS.
  • With a narrow focus, intervals were 65% shorter.
  • When a signal exists ($\mu^*=1$), FTS intervals remained significantly shorter (45% shorter with wide focus, 60% with narrow focus).
• Limit Setting: FTS provided tighter upper bounds when no signal was present and higher lower bounds when a signal was present, effectively improving the sensitivity to rule out or discover signals with less data.

Computational Efficiency

• The use of Quantile Regression to estimate critical values significantly reduced the simulation budget required. For a low budget of 9,000 pseudo-experiments, QR achieved Mean Squared Error (MSE) comparable to MC methods using 1.35 million (for LRS) and 30,000 (for FTS) pseudo-experiments.

5. Significance and Claims

The paper claims that the FTS offers a principled way to enhance the sensitivity of particle physics experiments, particularly in regimes characterized by low signal-to-noise ratios and small sample sizes (e.g., dark matter searches, neutrino oscillations, and rare Higgs decays).

• Flexibility: The method allows physicists to incorporate prior knowledge (e.g., theoretical predictions or previous measurements) into the statistical test via the focus function, without compromising the frequentist coverage of the results.
• Generality: The approach is applicable to any existing particle physics analysis as a "drop-in" replacement for the LRS and extends to high-dimensional or unbinned analyses where traditional MC belt constructions are computationally infeasible.
• Cautionary Note: The authors emphasize that while FTS introduces technical bias to optimize power, this does not equate to Bayesian inference or invalid coverage. They also caution that empty confidence sets (a known issue in HEP) may still occur and suggest reporting singleton point estimates or estimated sensitivities in such cases, rather than interpreting empty sets as meaningful exclusions.

The authors conclude that by focusing statistical power on regions of physical interest, the FTS can improve the discovery potential and precision of flagship measurements across the field.