⚛️ high-energy experiments

Coverage Is Not Ordering: Ancillary Leakage and Representation Dependence in Covariance-Based Inference

This paper demonstrates that replacing a full statistical model with a covariance matrix can alter the likelihood-ratio ordering and confidence decisions of an experiment by improperly reintroducing ancillary information into parameter inference, leading to significant discrepancies in coverage even when both methods are exactly calibrated.

Original authors: Tommaso Dorigo

Published 2026-09-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tommaso Dorigo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of fundamental science, researchers often measure the same thing multiple times, hoping that combining these observations will reveal a clearer picture of reality. When they do this, they usually report a central value—the best guess of the true number—and an uncertainty range that tells us how much that guess might wiggle. If several different quantities are measured together, scientists also provide a covariance matrix. Think of this matrix as a compact map that shows how the errors in one measurement might be linked to errors in another. For decades, this map has been treated as a reliable summary, a convenient shorthand that allows other scientists to reuse the data without needing the full, complex original model. The assumption has been that if you have the central values, the uncertainties, and this map of connections, you have everything you need to draw the same conclusions as the original experimenters.

However, a new analysis by physicist Tommaso Dorigo suggests that this convenient shorthand can sometimes lead us astray, not just by shifting a number slightly, but by fundamentally changing how we decide what is true. The study focuses on a specific type of measurement where a shared source of error affects all observations at once, like a single faulty scale weighing several different objects. In these situations, the data naturally split into two parts: the part that tells us about the true value we are looking for, and a separate part that simply records how much the individual measurements disagree with each other. In a perfect statistical world, the disagreement between measurements is just background noise; it contains no information about the true value and should be ignored when calculating the final result. Yet, when scientists replace the full statistical model with the simplified covariance map, this background noise can accidentally leak into the calculation, influencing the final answer in ways that the original model never intended.

The paper demonstrates that this leakage happens because the simplified map is often built using the observed data itself. When the measurements fluctuate up or down, the map changes to reflect those specific fluctuations. This creates a situation where the final conclusion depends not only on the average of the measurements but also on how much they disagree. To understand the severity of this, the researcher compared two ways of making a decision about a physical parameter. The first method uses the complete, exact statistical model, while the second uses the simplified covariance map. Both methods were carefully calibrated to be correct 68.27 percent of the time, meaning that if the experiment were repeated many times, both would include the true value in their reported range for roughly two-thirds of those trials. One might assume that if both are correct the same amount of the time, they are effectively doing the same job.

The study found that this assumption is false. Even though both methods accept the same total amount of probability, they accept different specific experiments. In a representative scenario where the total uncertainty was ten percent, the two methods disagreed on the outcome of about 7.6 percent of the repeated experiments. In other words, for nearly one in every thirteen trials, one method would say the data supports a specific value while the other method would reject it. This discrepancy is significant because it happens even when the overall success rate looks perfect. The problem is that the simplified method is sensitive to the "ancillary" disagreement—the noise that should have been ignored—whereas the exact method is not. This means that the simplified approach is effectively making decisions based on irrelevant fluctuations, a flaw that standard checks for accuracy often miss because they only look at the total success rate, not at which specific experiments are being accepted or rejected.

The research further shows that this issue is not limited to simple cases or specific types of data. It persists even when the measurements are expressed in different mathematical forms, such as converting a particle's lifetime into a decay width. While the exact statistical model remains consistent regardless of how the data is labeled, the simplified covariance approach can change its mind depending on the units used. The study also explored what happens when more than two measurements are combined. It turns out that the problem actually gets worse as more data is added, because the simplified method continues to let the internal disagreement of the group influence the final result. In fact, for three or more measurements, there are specific combinations of uncertainties where the standard accuracy checks show zero error, yet the simplified method still makes different decisions in a non-negligible fraction of cases. This creates a "blind spot" where the method appears flawless by conventional standards but is actually making inconsistent judgments.

The core finding is that a covariance matrix, while useful for summarizing linear relationships, is not a perfect substitute for the full statistical story. It can manufacture a false sense of relevance for data that should be ignored, leading to confidence intervals that shift based on how much the measurements disagree rather than on the measurements themselves. The paper concludes that relying solely on these simplified summaries can hide a fundamental instability in how scientific conclusions are drawn. To avoid this, the author suggests that scientists should preserve and publish the full statistical model whenever possible, rather than just the central values and the error map. This ensures that the method used to judge the data remains faithful to the original experiment, preventing the accidental introduction of bias from irrelevant fluctuations. The work does not claim that covariance matrices are useless, but it does prove that they are not a neutral replacement for the full model, and that their use can alter the very criteria by which experimental outcomes are judged.

Technical Summary: Coverage Is Not Ordering: Ancillary Leakage and Representation Dependence in Covariance-Based Inference

Problem Statement
Experimental measurements in fundamental science are frequently summarized by central values, uncertainties, and a covariance matrix. While these compact summaries are essential for subsequent combinations and reinterpretations, they discard information present in the original statistical model. This paper investigates which inferential properties are robust when a full statistical model is replaced by a covariance-based Gaussian reconstruction. Specifically, the author examines whether such approximations preserve not only fitted values and uncertainties but also the likelihood-ratio ordering of experimental outcomes.

The paper addresses two familiar scenarios where this issue arises:

Peelle's Pertinent Puzzle (PPP): Occurring when a shared multiplicative uncertainty is converted into an absolute covariance using observed central values. This can cause downward fluctuations to reduce the assigned covariance, artificially pulling the combined estimate downward.
Nonlinear Transformations: When measurements are expressed in nonlinear variables (e.g., converting a cross-section to a coupling strength), a propagated covariance used to reconstruct a Gaussian likelihood may fail to share the invariance properties of the exact likelihood.

The core question is whether these covariance approximations alter the ranking of possible experimental outcomes, thereby changing confidence decisions even when the procedures are separately calibrated to have identical coverage.

Methodology
The author analyzes a tractable correlated-measurement model involving two (and later $N$ ) measurements of a common positive quantity $\mu$ . The model includes:

Independent Gaussian statistical errors ( $\epsilon_i$ ).
A shared multiplicative random normalization ( $s\xi$ ).

The analysis proceeds by decomposing the observed data into two statistically independent components:

Informative Component ( $m$ ): A weighted mean containing all information about the parameter $\mu$ .
Ancillary Component ( $H$ ): A residual disagreement statistic (related to the chi-square goodness-of-fit) that describes the internal consistency of the measurements but contains no information about $\mu$ in the exact model.

The author compares the exact marginal likelihood (which depends only on $m$ ) against two common covariance-level reconstructions:

Data-Centered: The covariance is evaluated using the observed central values (the PPP prescription).
Model-Centered: The covariance is propagated from a candidate model value, often involving nonlinear transformations (e.g., Box-Cox family).

To quantify the discrepancy, the paper introduces $D_{ord}$ , defined as the probability mass of the symmetric difference between the acceptance regions of the exact and approximate constructions. $D_{ord}$ represents the fraction of repeated experiments for which the two methods make different "include/exclude" decisions regarding a tested parameter value.

Key Contributions and Results

Ancillary Leakage: The study demonstrates that covariance approximations can introduce a dependence on the ancillary statistic $H$ into the inference of $\mu$ . In the exact model, $H$ is ancillary and should not influence the estimate of $\mu$ . However, data-dependent covariances and nonlinear Gaussian reconstructions cause the fitted mean and the likelihood-ratio ordering to depend on the magnitude of the internal disagreement $H$ . This phenomenon is termed "ancillary leakage."
Ordering vs. Coverage Hierarchy: The paper establishes a critical hierarchy in the sensitivity of diagnostics to approximation errors:
- The ordering discrepancy ( $D_{ord}$ ) is generically first order in the total relative uncertainty ( $\lambda$ ): $D_{ord} = O(\lambda)$ .
- Conventional coverage diagnostics (fixed-threshold coverage differences and ancillary-conditioned coverage defects) begin at second order: $O(\lambda^2)$ .
- This implies that the ordering of experimental outcomes can change significantly even when the total probability content of the acceptance region appears unchanged to first-order coverage tests.
Coverage-Blind Points: The author identifies specific parameter configurations ("coverage-blind points") where the leading second-order coverage difference vanishes, yet the first-order ordering discrepancy remains non-zero.
- In a representative case with 10% total relative uncertainty, approximately 7.6% of experiments change their confidence decision between the exact and approximate methods, despite both being exactly calibrated to 68.27% coverage.
- This effect is shown to occur not only in nonlinear transformations but also in the standard PPP construction when $N \ge 3$ measurements are combined.
Generalization to Arbitrary $N$ : The analysis is extended to arbitrary finite numbers of measurements ( $N$ ) and arbitrary statistical variances. For the data-centered PPP prescription, the entire problem reduces to the weighted mean and a single goodness-of-fit statistic $H \sim \chi^2_{N-1}$ . The ordering discrepancy scales linearly with the number of residual degrees of freedom ( $N-1$ ).
Connection to Relevant Subsets: The mechanism is linked to the classical literature on ancillarity and relevant subsets. The covariance approximation effectively "manufactures" inferential relevance for a goodness-of-fit statistic that is ancillary in the true model. Consequently, data sets with the same informative weighted mean but different internal disagreements are treated differently, leading to different confidence intervals.

Significance and Claims
The paper argues that coverage alone is insufficient to certify that an approximate covariance likelihood preserves the intended likelihood-ratio ordering of the underlying measurement model.

Distinction: Coverage controls how much probability is accepted, while ordering specifies which experiments constitute that probability. Two procedures can have identical coverage but accept disjoint sets of experimental outcomes.
Implication: In high-energy physics and other fields relying on covariance matrices for combination and reinterpretation, the replacement of a full statistical model with a covariance-based Gaussian reconstruction can alter the fundamental criterion by which experimental outcomes are judged.
Recommendation: The author suggests that the strongest remedy is to preserve and publish the full statistical model (or simplified likelihoods) rather than relying solely on central values and covariance matrices. They emphasize that the ordering prescription (e.g., Feldman-Cousins) is not the source of the problem; rather, applying it to an approximate likelihood faithfully propagates the ordering distortions introduced by the approximation.

The work clarifies that while covariance matrices are faithful summaries for linear Gaussian measurements, they are not representation-invariant substitutes for full statistical models, particularly in the presence of correlated uncertainties and nonlinear transformations.

Technical Summary: Coverage Is Not Ordering: Ancillary Leakage and Representation Dependence in Covariance-Based Inference

More like this