⚛️ high-energy experiments

Silent coverage failures in rare-event searches and a degeneracy index that predicts them

This paper introduces the Poisson–Fisher degeneracy index to diagnose how specific model misspecifications in rare-event searches can cause silent coverage failures in frequentist intervals, demonstrating that certain biases evade standard goodness-of-fit tests and necessitate constrained nuisance parameters to restore statistical validity.

Original authors: Davide Pagno

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Davide Pagno

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery in a very quiet room. In the world of physics, this "quiet room" is a low-background experiment designed to find new, rare particles—like a ghostly dark matter particle bumping into an atom, or a mysterious double-beta decay event. The catch? These events are so rare that you might only see a handful of them, or maybe none at all, amidst a sea of ordinary noise. To make sense of this, scientists use a tool called a "confidence interval." Think of this as a safety net: if they say, "We are 90% confident the answer is between X and Y," that net should actually catch the true answer 90% of the time if you repeated the experiment a thousand times. This is called "coverage."

However, there's a sneaky problem. The safety net is built based on a map of the room (a mathematical model) that scientists think is perfect. But what if the map is slightly wrong? What if there's a hidden draft in the room (a background tail) or the floor is slightly tilted (a detector efficiency error) that the map doesn't show? If the map is wrong, the safety net might look perfect on paper but fail to catch the truth in reality. This is called "model misspecification," and it can lead to "silent coverage failures"—where the scientists think they are safe, but they aren't, and their tests don't even scream "Warning!" because the error is hiding in plain sight.

This paper, titled "Silent coverage failures in rare-event searches and a degeneracy index that predicts them," acts like a new kind of radar for these silent failures. The author, Davide Pagano and colleagues, wanted to figure out exactly how different types of map errors mess up the safety net. They didn't just look at the final answer; they looked at the two ends of the safety net separately: the "discovery" end (trying to prove something new exists) and the "exclusion" end (trying to prove something doesn't exist).

The researcher simulated various scenarios where the "map" was slightly off. They tested things like an unmodelled background tail (extra noise at low energies), a shifted dark matter halo (the source of the particles is moving differently than expected), or an overestimated signal efficiency (thinking the detector is better than it really is). They found that the direction of the error matters immensely. If the error looks like a "positive signal" (more events than expected), it tends to break the discovery side of the net, making scientists think they found a ghost when they didn't, while making the exclusion side (the upper limit) overly cautious. Conversely, if the error looks like a "negative signal" (fewer events than expected, perhaps because the detector is less efficient than thought), it breaks the exclusion side, making the upper limit too tight and potentially missing the truth.

To catch these errors before they cause trouble, the author introduced a new diagnostic tool called the "Poisson–Fisher degeneracy index," which they call $I_{PF}$ . You can think of this index as a two-part compass. The first part, $\beta$ , tells you how much the error will push your final answer in the wrong direction (the "damage"). The second part, $\gamma$ , tells you how obvious that error will be to a standard "goodness-of-fit" test (the "detectability"). The most dangerous situation, the author found, is when you have a large $\beta$ (big damage) but a tiny $\gamma$ (hard to detect). It's like a thief who steals a huge amount of money but leaves no fingerprints; the standard alarm system won't go off, but your bank account is in trouble.

The paper shows that in these dangerous "silent" cases, simply running the experiment again or checking the data with standard tools isn't enough. The solution, they suggest, is to admit the uncertainty. Instead of pretending the error doesn't exist, scientists should treat the potential error as a "nuisance parameter"—a known unknown that they constrain with extra information. In their simulations, when they did this (adding a constraint for the efficiency error, for example), the safety net was restored and started catching the truth again. However, there is a cost: the safety net becomes wider. The interval gets larger, meaning the scientists are less precise, but at least they are no longer lying about their confidence.

In short, this paper warns that in the hunt for rare physics events, a perfect-looking model can still be a trap. It provides a new way to predict when a model error will silently break the rules of statistics and offers a practical fix: if you suspect a hidden error that looks like a signal, don't ignore it. Model it, constrain it, and accept that your answer will be a bit fuzzier, but at least it will be honest.

Technical Summary: Silent Coverage Failures in Rare-Event Searches and a Degeneracy Index

Problem Statement
Searches for new physics in low-background experiments (e.g., dark matter direct detection and neutrinoless double-beta decay) often rely on inferring non-negative signal strengths from few events, resulting in upper limits. The validity of these frequentist inferences depends on coverage: the probability that a nominal confidence interval contains the true parameter value in repeated experiments.

Two primary challenges compromise coverage guarantees:

The Boundary: The parameter of interest ( $\mu \ge 0$ ) sits at the edge of its physical domain, violating regularity conditions required for standard asymptotic theory (e.g., Wilks' theorem).
Model Misspecification: The likelihood or simulator used for inference rarely matches the true data-generating process. Unmodeled features (e.g., low-energy background tails, uncertain halo models, or efficiency errors) can invalidate coverage guarantees.

The paper identifies a specific, dangerous failure mode: "silent" coverage failures. These occur when an interval misses its frequentist target (e.g., a 90% interval covers the truth less than 90% of the time) while the accompanying goodness-of-fit (GoF) test remains at its nominal false-positive rate. In such cases, the error is undetected by standard diagnostics.

Methodology
The author investigates this problem across three settings: an exact Poisson counting experiment, a dark-matter recoil spectral analysis, and a neutrinoless double-beta decay ( $0\nu\beta\beta$ ) peak search. They evaluate six interval construction procedures (Feldman–Cousins, CLs, profile likelihood, Bayesian, LF2I, and conformal-NPE) under controlled model deformations.

Deformations are modeled as additive changes to the expected count vector, $\delta\nu$ , representing sources like unmodeled background tails, shifted dark-matter halo parameters, or efficiency losses. The study distinguishes between point-wise coverage (conditional on a fixed true parameter) and marginal coverage (averaged over a prior).

To diagnose these failures, the author introduces the Poisson–Fisher Degeneracy Index, $I_{PF}(\delta\nu; \vartheta_0) = (\beta, \gamma)$ , calculated at a reference point $\vartheta_0$ :

$\beta$ (Signed Standardized Signal Bias): The projection of the deformation $\delta\nu$ onto the fitted tangent space (spanned by signal and nuisance parameters), expressed in profiled standard-error units. It predicts the shift in the fitted signal strength.
$\gamma$ (Orthogonal Residual): The Poisson–Fisher norm of the component of $\delta\nu$ orthogonal to the fitted tangent space. It measures the mismatch available to a saturated-Poisson goodness-of-fit test.

The index is derived using the local quadratic metric of the Poisson deviance and Fisher information, projecting the deformation through the complete fitted model (including nuisance parameters).

Key Contributions

The Poisson–Fisher Degeneracy Index ( $I_{PF}$ ): A two-coordinate diagnostic that maps a model deformation to potential endpoint damage ( $\beta$ $β$ ) and GoF detectability ( $\gamma$ $γ$ ).
- The sign of $\beta$ identifies which endpoint is threatened (positive $\beta$ threatens the lower/discovery endpoint; negative $\beta$ threatens the upper/exclusion endpoint).
- The magnitude of $\gamma$ indicates whether the error is detectable by standard GoF tests.
Controlled Severity Protocol: A systematic evaluation of how coverage degrades as model deformations increase, separating lower-endpoint, upper-endpoint, and whole-interval coverage.
Coverage–Sensitivity Trade-off Analysis: A demonstration that promoting a plausible, signal-like deformation to a constrained nuisance parameter (using auxiliary information) can restore coverage, albeit at a quantifiable cost in interval sensitivity (wider intervals).
Empirical Assessment: Validation of the index across exact counting, dark-matter spectra, and $0\nu\beta\beta$ peak searches, showing that spectral information only aids detection when the deformation is identifiable against the full fitted model.

Results

Directional Bias: Positive signal-like biases (e.g., unmodeled background) degrade lower-endpoint coverage (leading to false discoveries) but make upper limits conservative. Conversely, negative biases (e.g., overestimated efficiency) can cause upper endpoints to undercover (anti-conservative limits).
Silent Failures: Deformations that are nearly collinear with the signal template (e.g., a background tail indistinguishable from signal in a single bin, or a halo shift absorbed by WIMP mass fitting) result in large $|\beta|$ but small $\gamma$ . These cases evade GoF detection while severely compromising coverage.
Nuisance Remedy: In an exactly collinear benchmark, modeling the deformation as a constrained nuisance restored coverage to nominal levels at evaluated grid points. However, this came with a sensitivity cost, increasing the mean interval length by up to a factor of 1.6 as auxiliary constraints weakened.
Spectral Resolution: In spectral analyses, the fitted mass parameter can absorb shape deformations (reducing $\beta$ and $\gamma$ ), shielding the endpoint from the full bias but also hiding the error from diagnostics. This confirms that single-template calculations overstate both damage and detectability.
Method Robustness: Simulation-based inference methods (LF2I, conformal-NPE) and asymptotic methods all lose coverage when data is generated outside the nominal model. Internal checks (like LF2I's nominal simulator checks) do not detect misspecification relative to the true data-generating process.

Significance and Claims
The paper claims that calibration under a nominal simulator does not protect against misspecification relative to the data-generating process. The primary contribution is the $I_{PF}$ index, which serves as a pre-toy screening tool to identify "dangerous" deformations (large $|\beta|$ , small $\gamma$ ) that require explicit treatment.

The author argues that for such deformations, analysts should not rely solely on GoF tests. Instead, plausible large-bias, low-residual deformations must be represented as nuisance parameters constrained by auxiliary information or included in a defensible envelope. Without such constraints, signal and deformation are not separately identifiable, and finite model-dependent limits should not be presented as robust. The study emphasizes that reporting robustness requires evaluating both lower and upper endpoints separately and acknowledging the trade-off between coverage validity and interval sensitivity.

Technical Summary: Silent Coverage Failures in Rare-Event Searches and a Degeneracy Index

More like this