⚛️ phenomenology

Testing the statistical compatibility of data sets with a common prior

This paper extends the Parameter Goodness-of-fit (PG) test to handle scenarios where data sets are constrained by a common prior, demonstrating that under a Gaussian approximation, the resulting test statistic follows a generalized $\chi^2$ distribution suitable for calculating p-values.

Original authors: Pilar Coloma, Michele Maltoni, Thomas Schwetz

Published 2026-10-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pilar Coloma, Michele Maltoni, Thomas Schwetz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of fundamental physics, researchers often face a puzzle that looks like a jigsaw with pieces from different boxes. They have data from one experiment, data from another, and perhaps a third piece of information from a previous study or a theoretical expectation. The goal is to see if all these pieces fit together to form a single, coherent picture of how the universe works. To do this, scientists use a statistical tool called a "goodness-of-fit" test. Think of it as a ruler that measures how well a proposed theory matches the actual numbers collected in the lab. If the numbers match the theory closely, the ruler shows a small error; if they clash, the error is large. A specific version of this ruler, known as the Parameter Goodness-of-fit test, has long been the standard for checking if two different sets of data agree with each other, provided they are completely independent of one another.

However, the real world of physics is rarely so clean. Often, the data sets being compared are not truly independent because they share a common constraint. Imagine two different teams measuring the same physical quantity, but both teams must also account for a shared source of uncertainty, like a known limit on a particle's behavior derived from a separate, older experiment. In this scenario, the two new data sets are linked by that shared history. The standard ruler breaks down here because it assumes the data sets are strangers to each other. When they are actually related through a common constraint, the standard test can give a misleading answer, potentially telling scientists that their data agrees when it does not, or vice versa. This is a critical problem in fields like neutrino physics and cosmology, where combining results from multiple sources is essential for discovering new laws of nature.

A team of physicists at the Instituto de Física Teórica in Madrid and the Karlsruhe Institute of Technology in Germany has developed a new way to handle this specific problem. They have extended the standard ruler to work even when the data sets are tied together by a shared constraint, which they call a "prior." Instead of forcing the data into an old framework that no longer fits, they designed two new versions of the test statistic. One version, which they call PG-star, allows the result to be negative, reflecting the complex way the data and the constraint interact. The other version, PG-plus, is built to always stay positive, ensuring the result behaves more like a traditional measurement of error. Both versions are designed to tell researchers exactly how likely it is that their data sets are compatible, even when they are influenced by the same external information.

The researchers did not just propose these new tools theoretically; they tested them rigorously using computer simulations. They created a virtual laboratory that mimicked the conditions of a real neutrino experiment, generating thousands of fake data sets that included statistical noise and the shared constraints found in real life. In these simulations, they applied both the new tests and the old standard test to see how they performed. The results were clear: the new tests matched the theoretical predictions perfectly. When the shared constraint was very strong, the new tests smoothly turned into the standard test, confirming that the new method is a true generalization of the old one. When the constraint was weak, the new tests correctly identified that the data sets were no longer independent, adjusting their calculations to reflect the true statistical relationship between the measurements.

One of the most interesting findings was how the two new tests behaved differently depending on where the disagreement came from. If the tension or conflict between the data sets arose from the shared constraint itself, the PG-plus test was slightly more sensitive to it. If the conflict came from the data sets disagreeing with each other while the constraint remained calm, the PG-star test offered a different perspective. By comparing the results of these two tests, scientists can now diagnose the source of a problem. If the two tests give very different answers, it suggests that the issue lies with the shared constraint. If they agree, the problem is likely within the data sets themselves. This diagnostic capability adds a new layer of clarity to how researchers interpret conflicting results.

The work also revealed that the new tests follow a specific mathematical pattern known as a generalized distribution. While this sounds complex, it simply means that the range of possible outcomes for the test is broader and more flexible than the standard bell curve used in older methods. The researchers provided a computer script that allows anyone to calculate the exact probability of their results, making the new method accessible for immediate use. In the end, this research offers a more robust way to combine information. It ensures that when scientists from different experiments come together to build a picture of the universe, they are using a ruler that accounts for the fact that they are all standing on the same ground. This allows them to trust their conclusions more deeply, knowing that the statistical tools they use have been updated to match the complexity of the data they collect.

Technical Summary: Testing the Statistical Compatibility of Data Sets with a Common Prior

Problem Statement
In fundamental physics, a frequent statistical challenge involves testing the compatibility of multiple data sets within a single theoretical model. The Parameter Goodness-of-fit (PG) test, originally introduced in Ref. [1], provides a frequentist framework for this purpose. However, the standard PG test relies on the assumption that the data sets being compared are statistically independent.

In many practical scenarios, this independence is violated because the model parameters are constrained by a common "prior" (an auxiliary data set or external knowledge). This prior often constrains nuisance parameters (e.g., systematic uncertainties like flux or cross-section normalizations) or specific model parameters. When a common prior is imposed, the data sets become statistically correlated through the shared fluctuations of the prior. Consequently, the standard PG test statistic, which follows a $\chi^2$ distribution under the assumption of independence, is no longer valid. The paper addresses the need for a generalized test statistic capable of handling this prior-constrained scenario.

Methodology and Theoretical Framework
The authors propose two modified test statistics, denoted as $\text{PG}^*$ and $\text{PG}^+$ , to evaluate the compatibility of two data sets ( $A$ and $B$ ) subject to a common prior ( $P$ ). The analysis is conducted within a frequentist framework using $\chi^2$ functions, assuming a Gaussian approximation near the $\chi^2$ minima.

Definitions:
- $\text{PG}^*$ : Defined as the difference between the global minimum of the combined $\chi^2$ (including the prior) and the sum of the individual minima of the data sets (including the prior). Unlike the standard PG, this quantity is not strictly positive and can assume negative values.
- $\text{PG}^+$ : Defined as the sum of the $\Delta\chi^2$ values for each data set relative to the global best fit. This construction ensures the statistic is strictly non-negative.
Distribution Derivation:
The authors derive the analytic distributions for these statistics under the Gaussian approximation. They demonstrate that neither statistic follows a simple $\chi^2$ distribution. Instead, they follow a generalized $\chi^2$ distribution, which is a linear combination of independent $\chi^2$ variables with specific coefficients.
- For $\text{PG}^*$ , the distribution is derived using the theory of quadratic forms. In the case of one model parameter ( $p=1$ ) and one nuisance parameter ( $q=1$ ), the statistic is shown to be distributed as $X_0 + \rho_{AB}(X_1 - X_2)$ , where $X_i$ are independent $\chi^2_1$ variables and $\rho_{AB}$ is a correlation parameter determined by the prior variance relative to the data variances.
- For $\text{PG}^+$ , the distribution is similarly derived, involving the eigenvalues of a matrix constructed from the correlation vectors of the discrepancies.
- The authors provide the characteristic functions for both statistics, allowing for the numerical calculation of Probability Distribution Functions (PDFs) and Cumulative Distribution Functions (CDFs) to obtain $p$ -values.
Limiting Cases:
- Strong Prior Limit: When the prior constraint is much stronger than the data constraints ( $s_P \to 0$ ), both statistics converge to the standard PG test for the unconstrained model parameters, following a standard $\chi^2_p$ distribution.
- Weak Prior Limit: When the prior is very weak ( $s_P \to \infty$ ), the behavior differs. $\text{PG}^*$ does not converge to a standard $\chi^2$ but remains a generalized distribution. In contrast, $\text{PG}^+$ converges to the standard PG test for the unconstrained data sets (distributed as $\chi^2_{p+q}$ ).

Key Contributions

Generalization of the PG Test: The paper extends the PG test to scenarios involving common priors, a situation where standard methods fail due to induced correlations.
Two Variants: The introduction of $\text{PG}^*$ and $\text{PG}^+$ offers flexibility. $\text{PG}^+$ is strictly positive and converges to standard limits in both strong and weak prior regimes, while $\text{PG}^*$ provides a different weighting of the prior term.
Analytic Solutions: The derivation of the generalized $\chi^2$ distributions and their characteristic functions allows for exact $p$ -value calculation without relying on computationally expensive Monte Carlo simulations for every new dataset configuration.
Diagnostic Tool: The authors show that comparing the $p$ -values of $\text{PG}^*$ and $\text{PG}^+$ can help diagnose the origin of tension. Since $\text{PG}^+$ places more weight on the prior term, a significant discrepancy between the two $p$ -values can indicate whether the tension arises primarily from the prior-data mismatch or from the data-data mismatch.

Numerical Results and Validation
The theoretical derivations were validated using Toy Monte Carlo simulations inspired by neutrino oscillation disappearance measurements.

Null Hypothesis: Simulations confirmed that the numerical distributions of $\text{PG}^*$ and $\text{PG}^+$ agree excellently with the derived analytic generalized $\chi^2$ distributions.
Alternative Hypotheses: The authors tested scenarios where the data sets and/or the prior were generated with conflicting true parameter values (tension). The tests successfully detected this tension, with the median sensitivity (power of the test) quantified for both statistics.
Comparison: Under the null hypothesis, it is more likely to obtain a larger $p$ -value for $\text{PG}^+$ than for $\text{PG}^*$ . This is attributed to the construction of $\text{PG}^+$ , which makes it less sensitive to statistical fluctuations in the individual data sets compared to the prior term.

Significance and Claims
The paper claims to provide a robust, analytically tractable method for assessing the compatibility of data sets when a common prior is present. The significance lies in:

Correcting a Statistical Flaw: It resolves the issue of applying the standard PG test to correlated data sets, which would otherwise yield incorrect $p$ -values.
Practical Utility: By providing a Python script (Ref. [12]) to calculate the distributions based on covariance matrices, the authors offer a straightforward recipe for practitioners to compute $p$ -values in the Gaussian limit.
Diagnostic Capability: The ability to distinguish the source of tension (prior vs. data) via the comparison of $\text{PG}^*$ and $\text{PG}^+$ $p$ -values is presented as a valuable diagnostic tool for model validation.

The authors remain modest regarding the scope, noting that their results are derived under the Gaussian approximation and assuming linear parameter dependence. They acknowledge that a detailed study with real-world neutrino data, where non-linearities and boundaries may be significant, is a subject for future work (Ref. [19]). The paper does not propose new experimental setups but rather a statistical framework applicable to existing or future analyses in fields such as neutrino physics and cosmology.

More like this