Hypothesis tests and model parameter estimation on data sets with missing correlation information

This paper proposes robust test statistics and an algorithm for inflating variances to enable conservative hypothesis testing and model parameter estimation on normally distributed data when full covariance information is unavailable.

The Gist

Imagine you are a detective trying to solve a mystery using clues from three different witnesses.

The Ideal Scenario:
In a perfect world, every witness gives you their testimony, and you also get a "relationship map" showing how they influenced each other. Maybe Witness A and Witness B are friends who talk to each other, so their stories are correlated. Witness C is a stranger. If you have this map, you can weigh the clues perfectly to find the truth.

The Real-World Problem:
In science (and in this detective story), we often don't get the relationship map.

• Witness A gives a report with a list of numbers and a "confidence interval" (how sure they are).
• Witness B gives a similar report.
• But neither tells you if they talked to each other. Did they copy each other? Did they share a source of error? We don't know.

If you just mash these reports together assuming they are totally independent (like strangers), you might think you have a super-precise answer. But if they were actually correlated (like friends copying each other), your "super-precise" answer is actually a lie. You are overconfident, and you might draw the wrong conclusion.

This paper by Lukas Koch is a guide on how to be a cautious detective when you don't have the full relationship map.

Here is the breakdown of the paper's two main solutions, using simple analogies:

---

Part 1: The "Worst-Case" Test (For Simple Checks)

The Goal: You just want to know: "Is this suspect (a scientific model) guilty or innocent?" You aren't trying to calculate their exact height or weight yet; you just want to know if they fit the crime scene.

The Problem: If you ignore the missing relationship map, you might think the evidence is overwhelming (e.g., "99.9% chance of guilt!"). But if the witnesses were actually colluding, that evidence might only be 60% strong. You've been tricked by false precision.

The Solution: The "Fitted" Test Statistic
Instead of trying to average all the clues together (which is dangerous if they are correlated), the author suggests a new rule: "Look at the single worst clue."

• The Analogy: Imagine you have three witnesses.
  • Witness 1 says: "The suspect is 90% likely to be guilty."
  • Witness 2 says: "The suspect is 95% likely to be guilty."
  • Witness 3 says: "The suspect is 99% likely to be guilty."
• The Old Way: You might average these and say, "Wow, 94.6%!" (Dangerous if they are all lying together).
• The New Way: You look at the maximum discrepancy. You say, "Okay, the strongest evidence against the suspect is 99%. Let's assume the worst-case scenario where all three witnesses are perfectly aligned in their error."

By focusing only on the "worst" single piece of evidence and ignoring the rest, you create a test that is conservative. It might not be the most sensitive test (it might miss some guilty people), but it will never falsely accuse an innocent person because of hidden correlations. It's like a safety net that ensures you don't make a mistake by being too confident.

The paper also introduces a "p-min" method, which is even simpler: Just take the smallest p-value (the strongest evidence) from all your tests and multiply it by the number of tests. It's a quick, dirty, but safe way to combine results.

---

Part 2: The "Inflation" Factor (For Fitting Models)

The Goal: Now you want to do more than just check guilt. You want to fit a model. You want to say, "The suspect is guilty, and their height is exactly 5'10" with an uncertainty of +/- 1 inch."

The Problem: The "Worst-Case" test from Part 1 is too blunt for this. It's like trying to measure a person's height with a sledgehammer. It's not smooth, and it doesn't give you a nice curve to work with. You need a smooth curve to find the "best fit."

The Solution: The "Derating" (or Inflation) Factor
Since we can't know the hidden correlations, the author suggests we pretend our measurements are less precise than they actually are. We artificially "inflate" the uncertainty.

• The Analogy: Imagine you are measuring a table with a ruler.
  • Normal situation: You measure it, and you are 95% sure it's 100cm long. Your uncertainty is +/- 1cm.
  • The "Missing Map" situation: You suspect your ruler might be slightly bent, or the table might be wobbly, but you don't know how.
  • The Fix: The author says, "Let's just assume your ruler is actually twice as wobbly as you thought." So, instead of saying 100cm +/- 1cm, you say "100cm +/- 2cm."

By making the "error bars" (uncertainties) wider, you ensure that even if the worst-case hidden correlations exist, your answer is still correct. You are trading precision for safety.

How much do we inflate?
The paper provides a clever algorithm (a step-by-step computer recipe) to calculate exactly how much to inflate the error bars.

1. It looks at the structure of your data (how many blocks of information you have).
2. It simulates the "Nightmare Scenario": What if every possible hidden correlation was 100% negative or positive?
3. It calculates the "Derating Factor" (e.g., 1.8 or 2.0).
4. You multiply your error bars by this factor.

Real World Example from the Paper:
The author applied this to neutrino physics (subatomic particles).

• They combined data from three different experiments (T2K, MINERvA, MicroBooNE).
• Without this method, the scientists thought they knew the parameters of the neutrino model very precisely.
• With the method: They realized that because they didn't know how the experiments were correlated, they had to inflate their uncertainties by up to 2x.
• The Result: The "best fit" point (the center of the answer) didn't change, but the "cloud" of uncertainty around it got much bigger. This is honest science. It says, "We think the answer is X, but because we don't know how these experiments talk to each other, we can't be as sure as we thought."

---

Summary: The Takeaway

1. Don't ignore the missing map: If you combine data from different sources without knowing how they relate, you risk being dangerously overconfident.
2. For simple "Yes/No" questions: Use the "Fitted" or "p-min" test. These look at the strongest piece of evidence and assume the worst-case correlation. They are safe and conservative.
3. For "How much?" questions (Fitting): Don't try to guess the correlations. Instead, use the author's algorithm to calculate an Inflation Factor. Multiply your error bars by this factor.
4. The Philosophy: It is better to be slightly less precise but honest about your uncertainty, than to be very precise but wrong because you ignored hidden connections.

The paper essentially gives scientists a "safety helmet" for when they are forced to work with incomplete information. It ensures that even in the worst-case scenario of hidden correlations, their conclusions remain valid.

Technical Summary

1. Problem Statement

In statistical analyses of normally distributed data, the standard practice is to utilize the full covariance matrix to account for correlations between data points. However, in many practical scenarios—such as combining results from separate publications or analyzing published results that omit covariance matrices—the full correlation information is unavailable.

• The Consequence: Ignoring unknown correlations leads to "undercoverage," where confidence intervals are too narrow, and p-values are artificially small, resulting in false exclusions of models or overconfident parameter estimates.
• The Gap: Existing methods either assume independence (risky) or use ad-hoc variance inflation (e.g., doubling variance) that lacks a rigorous theoretical basis for specific confidence levels. There is a need for robust statistical tools that remain conservative (i.e., do not overstate significance) in the presence of unknown correlations.

2. Methodology

The paper proposes two distinct approaches depending on the statistical goal: Simple Hypothesis Testing and Model Parameter Estimation.

A. Robust Test Statistics for Simple Hypothesis Tests

For testing whether a specific model (with fixed parameters) fits the data, the author generalizes the "fitted" test statistic introduced in previous work.

• The "Fitted" Statistic: Instead of minimizing the Mahalanobis distance ($M^2$) over a known covariance, this method treats unknown off-diagonal covariance elements as nuisance parameters. It minimizes the $M^2$ over the space of all possible valid covariance matrices.
• Result: The minimum possible $M^2$ is mathematically equivalent to the maximum of the squared $M$-distances of the individual known covariance blocks.
  $$\text{fitted}(x|\mu, S) = \max_i \left( (x_i - \mu_i)^T S_{ii}^{-1} (x_i - \mu_i) \right)$$
• Distribution: The distribution of this statistic is derived as the product of the Cumulative Distribution Functions (CDFs) of the individual blocks, termed the "Cee-squared" distribution.
• Optimization ($f_{max}$): The paper generalizes this to any statistic that is the maximum of strictly increasing functions of block $M$-distances. Two specific variants are proposed:
  1. $p_{min}$: Takes the minimum p-value among blocks.
  2. Optimal-$f_{max}$: Uses a function derived from the ratio of the $\chi^2$ CDF to its PDF to maximize statistical power while maintaining robustness.

B. Derating Factor for Parameter Estimation

For fitting model parameters (where parameters are free variables), the "fitted" statistic is unsuitable because it is non-smooth and lacks a known distribution for parameter differences (no Wilks' theorem equivalent).

• The Approach: The author proposes inflating the parameter uncertainties by a constant derating factor ($\alpha$). This ensures that the confidence intervals remain conservative up to a chosen confidence level (e.g., 3$\sigma$) regardless of the unknown correlations.
• Algorithm for $\alpha$:
  1. Whitening: Transform data blocks into a standard normal form using block-wise whitening matrices.
  2. Worst-Case Construction: An algorithm iteratively sets unknown off-diagonal elements of the whitened covariance matrix to $\pm 1$ (the "nightmare scenario") to maximize the variance of the test statistic. It selects elements that maximize the trace of the projection matrix product.
  3. Calculation: The factor $\alpha$ is the ratio of the quantile (Inverse Distribution Function) of the test statistic under this "nightmare" covariance to the quantile under the assumption of no correlations.
• Approximation: An empirical approximation for $\alpha$ is provided based on the total number of bins ($k$) and the average block size, which is generally smaller than the "double variance" heuristic but sufficient for high confidence levels.

C. Goodness of Fit (GoF) and Composite Hypotheses

The derating method is extended to GoF tests. By using the "residual maker" matrix (which projects data onto the null space of the model parameters) instead of the model Jacobian, the same worst-case covariance algorithm determines a derating factor for GoF statistics, ensuring conservative p-values.

3. Key Contributions

1. Generalized Fitted Statistic: A rigorous derivation showing that the robust test statistic for unknown correlations between blocks is the maximum of the block-wise $M$-distances.
2. Algorithmic Derating Factor: A novel algorithm to calculate the specific variance inflation factor required to guarantee conservative coverage for parameter fits and GoF tests, replacing arbitrary heuristics.
3. Optimal-$f_{max}$ Statistic: A new test statistic designed to maximize statistical power for simple hypothesis tests while maintaining robustness against unknown correlations.
4. Software Implementation: The methods are implemented in the open-source Python package NuStatTools.

4. Results and Applications

The paper validates the methods using toy data and real-world neutrino interaction data:

• Toy Data: Simulations with 10-dimensional Gaussian data show that naive $M$-distance tests suffer severe undercoverage (real significance is much weaker than assumed) when correlations are ignored. The proposed "fitted" statistic and the inflated parameter uncertainties remain consistently conservative across all correlation levels (up to $\rho=0.99$).
• Neutrino Model Tuning (T2K, MicroBooNE, MINERvA):
  • Hypothesis Testing: When combining multiple neutrino cross-section measurements, the $p_{min}$ and optimal-$f_{max}$ statistics allow for quantitative statements about model exclusion (e.g., excluding the Spectral Function model at 98% CL) that were previously impossible without full covariance data.
  • Parameter Fitting: Applying the derating algorithm to a GENIE neutrino event generator tune (fitting "RedPar" parameters) yielded a derating factor between 2.70 and 3.87 (depending on assumptions about inter-experiment correlations). This implies parameter uncertainties must be inflated by a factor of 1.64 to 1.97 to remain conservative.
  • GoF: The method successfully produces conservative GoF p-values for composite hypotheses.

5. Significance

This work provides a critical toolkit for modern high-energy physics and data science, where combining heterogeneous datasets is common but full covariance information is often lost.

• Conservatism: It prevents the scientific community from drawing false conclusions (false exclusions) due to ignored correlations.
• Rigorous vs. Ad-hoc: It replaces arbitrary "variance doubling" with a mathematically derived factor based on the specific geometry of the data blocks and the model subspace.
• Applicability: The methods are applicable to any field dealing with multivariate normal data where correlation structures are partially unknown, particularly in meta-analyses of experimental results.

In summary, Koch presents a comprehensive framework to handle missing correlation data, offering robust alternatives for hypothesis testing and a rigorous algorithm for inflating uncertainties in parameter estimation, ensuring that statistical conclusions remain valid even in the worst-case correlation scenarios.