Confidence, Statistical Evidence and Relative Belief with Applications to a Problem in Particle Physics

This paper applies relative belief inferences, which satisfy both Bayesian likelihood ordering and frequentist confidence requirements, to construct uncertainty intervals for a Poisson signal-with-background model in particle physics, demonstrating their advantages over the standard Feldman-Cousins approach.

The Gist

Imagine you are a detective trying to solve a mystery in a very noisy room. The "mystery" is whether a new, rare particle has been created in a physics experiment. The "noise" is the background radiation that is always there, even when nothing new is happening.

This paper, written by Michael Evans and Siqi Zheng, is about how to tell the difference between a real discovery and just random noise, and how to measure how sure we can be about that answer.

Here is the breakdown of their argument using simple analogies:

1. The Goal: Finding the Signal in the Noise

In particle physics, scientists count events. Sometimes they see a lot of events. Is it because a new particle was found (the Signal), or just because the background noise got louder (the Background)?

The authors argue that the main job of statistics isn't just to give a number; it's to reveal evidence. They ask: Does the data actually point toward a new particle, or is it just a fluke?

2. The Old Way: The "Feldman-Cousins" Interval

For a long time, physicists have used a method called the Feldman-Cousins Confidence Interval (FCCI).

• The Analogy: Imagine you are trying to guess the weight of a hidden object. The FCCI is like a safety net. It says, "If we repeated this experiment 100 times, 95 of those nets would catch the true weight."
• The Problem: The authors argue that while this net is good for catching the truth over the long run, it doesn't always tell you what the current data is actually saying.
  • Sometimes, the net includes weights that the data actually says are unlikely (violating the "likelihood ordering").
  • Sometimes, the net behaves strangely. For example, if you see zero events, the FCCI might get smaller if you assume the background noise is higher. The authors say this makes no sense: if you see nothing, your uncertainty about the new particle shouldn't shrink just because you think the background is louder.

3. The New Way: "Relative Belief" and the "Plausible Region"

The authors propose a different approach called Relative Belief.

• The Analogy: Imagine you have a hunch (a Prior) about where the new particle might be. Then, you get new data (the Evidence).
  • Relative Belief asks: "How much did my hunch change after seeing the data?"
  • If the data makes a specific value much more likely than it was before, that's evidence in favor.
  • If the data makes a value much less likely, that's evidence against.
• The Plausible Region: This is the authors' new "interval." It is a list of all the values that the data has boosted in our belief.
  • Think of it as a "Shortlist of Suspects." The Plausible Region only includes suspects that the evidence has made more likely than they were before the investigation started.
  • If a suspect is on the list, the data supports them. If they aren't, the data doesn't support them.

4. Why the New Way is Better (According to the Paper)

The authors claim the Plausible Region is superior for science for three main reasons:

1. It Respects the Evidence: The Plausible Region is always a "Likelihood Region." This means it never includes a value that the data says is less likely than another value outside the region. The old FCCI sometimes breaks this rule.
2. It Avoids Absurdity: The old FCCI can sometimes produce a result that covers every possible value (the whole parameter space). The authors say this is silly because if you say "it could be anything," you haven't learned anything. The Plausible Region never does this; it always narrows things down based on what the data actually supports.
3. It Handles Noise Better: In their examples, when the background noise is high or unknown, the Plausible Region stays stable and logical. The FCCI, however, can behave erratically (like shrinking when it shouldn't).

5. Checking the Work: "Bias" and "Reliability"

The authors know that scientists worry about reliability (Frequentist concerns). They don't just say, "Trust our math." They also run "Bias Checks."

• The Analogy: Before you go on a fishing trip, you check your boat to make sure it won't sink.
• The Check: They calculate, before doing the experiment, how often their method might fail.
  • Bias Against: How often do we miss a real discovery?
  • Bias In Favor: How often do we claim a discovery when there isn't one?
• They show that by choosing the right amount of data (sample size), they can make these errors very small, ensuring their "Plausible Region" is reliable, just like the old methods, but without the logical flaws.

6. Real-World Test: The Neutrino Experiment

The paper tests this on a real historical experiment (Karmen II) where scientists were looking for neutrino oscillations.

• The Result: In the first part of the experiment, the data was weak, and the results depended heavily on initial guesses. But as more data came in, the "Plausible Region" stabilized and gave a clear answer: There was no evidence for a signal.
• The authors note that their method handled the "background noise" (which was uncertain) much more naturally than the old methods could.

Summary

The paper argues that while the old "Confidence Interval" method is good for long-term error rates, it often fails to accurately represent what the current data is telling us.

The authors propose Relative Belief as a better tool. It creates a Plausible Region that strictly follows the logic of the evidence: it only includes values that the data has made more believable. They prove that this method is not only logically sound but also reliable enough to satisfy strict scientific standards, making it a better way to report discoveries in particle physics.

Technical Summary

Technical Summary: Confidence, Statistical Evidence and Relative Belief with Applications to a Problem in Particle Physics

Problem Statement
The paper addresses the fundamental difficulty in statistical analysis of defining and quantifying "statistical evidence," particularly in the context of particle physics experiments involving Poisson-distributed counts with background noise. The authors critique the prevailing use of Feldman-Cousins Confidence Intervals (FCCI) and other frequentist confidence regions. While these methods satisfy repeated sampling (frequentist) coverage requirements, the authors argue they fail to properly represent statistical evidence. Specifically, FCCIs can violate the likelihood ordering (excluding parameter values with higher likelihood than those included) and can produce "improper" regions (e.g., covering the entire parameter space or excluding values supported by the data) when parameters are constrained (e.g., $\lambda \ge 0$). The core problem is reconciling the evidential goal of revealing what data indicates with the behavioristic goal of ensuring inference reliability under repeated sampling.

Methodology: Relative Belief Inference
The authors propose and apply Relative Belief Inference, a Bayesian framework grounded in the Principle of Evidence. This principle states that evidence in favor of a hypothesis $H$ exists if the posterior probability exceeds the prior probability ($P(H|data) > P(H)$), and evidence against exists if the posterior is lower.

Key methodological components include:

1. Relative Belief Ratio (RB): Defined as $RB(\psi | x) = \frac{\pi(\psi|x)}{\pi(\psi)} = \frac{m(x|\psi)}{m(x)}$, where $\pi$ is the prior, $\pi(\cdot|x)$ is the posterior, and $m$ is the marginal likelihood.
   • $RB > 1$: Evidence in favor.
   • $RB < 1$: Evidence against.
   • $RB = 1$: No evidence either way.
2. Plausible Region: The set of parameter values where $RB > 1$. This region is guaranteed to be a likelihood region (respecting the likelihood ordering) and contains all values with evidence in their favor.
3. Estimation: The relative belief estimate is the value maximizing the RB, which coincides with the Maximum Likelihood Estimate (MLE) under the marginal model.
4. Bias Calculations: To address frequentist reliability, the authors employ a priori bias calculations:
   • Bias Against: The prior probability of failing to find evidence in favor of a true value (Type I error analogue).
   • Bias in Favor: The prior probability of finding evidence in favor of a meaningfully false value (Type II error analogue).
     These are used in experimental design to select sample sizes that ensure reliable inferences.
5. Prior-Data Conflict Checking: The methodology includes a check (Evans and Moshonov, 2006) to ensure the prior does not place the true parameter in the tails of the prior distribution relative to the observed data. If conflict is detected, the prior is modified.

Application to Particle Physics
The methodology is applied to the problem of detecting a new particle signal ($\lambda$) amidst background noise ($b$), modeled as $X \sim \text{Poisson}(\lambda + b)$. Two scenarios are analyzed:

1. Known Background ($b$ is known): A Gamma prior is placed on $\lambda$. The plausible interval is constructed, and its frequentist coverage and bias properties are evaluated.
2. Unknown Background ($b$ is unknown): Independent Gamma priors are placed on both $\lambda$ and $b$. The nuisance parameter $b$ is integrated out to form a marginal model for $\lambda$. The same relative belief framework is applied.

Key Results

• Violation of Likelihood Ordering by FCCI: The paper demonstrates via examples (including discrete models and normal means) that FCCIs often violate the likelihood ordering. For instance, an FCCI may exclude a parameter value $\theta_3$ while including $\theta_2$, even when the likelihood of the data is higher under $\theta_3$ than $\theta_2$.
• Properness of Plausible Regions: Unlike FCCIs, plausible regions derived from relative belief are never equal to the entire parameter space (unless the likelihood is flat, in which case the region is empty). They strictly adhere to the likelihood ordering.
• Performance Comparison:
  • In simulations with known background, the plausible interval achieves frequentist confidence levels comparable to FCCIs (e.g., >90% for $n=10$) while maintaining the property of being a likelihood region.
  • The plausible interval exhibits significantly lower "bias in favor" (probability of covering meaningfully false values) compared to FCCIs across various sample sizes and meaningful difference thresholds ($\delta$).
  • FCCIs exhibit sensitivity to the background rate $b$ when zero events are observed (the upper limit decreases as $b$ increases), a behavior the plausible interval avoids.
• Real-World Application (Karmen II): The method was applied to Karmen II neutrino oscillation data. Using a sequential Bayesian strategy, the plausible interval robustly stabilized after the second experiment, confirming strong evidence for the null signal ($\lambda=0$) regardless of initial prior assumptions. The authors note that a direct comparison with FCCI is structurally inappropriate here due to the sequential nature of the data and the treatment of $b$ as a nuisance parameter.

Significance and Claims
The paper claims that relative belief inferences offer a more appropriate framework for scientific contexts than traditional confidence regions because they directly address the definition of evidence.

• Evidence vs. Error: The authors argue that while confidence regions are designed to measure error rates (behavioristic), they do not necessarily reflect evidence. Relative belief regions satisfy the Principle of Evidence (Theorem 1), ensuring that any reported interval respects the likelihood ordering.
• Integration of Approaches: The methodology successfully combines the evidential approach (inference based on belief change) with the behavioristic approach (design based on bias control). The a priori bias calculations ensure that the resulting inferences are reliable under repeated sampling, satisfying frequentist requirements without sacrificing the coherence of the evidential interpretation.
• Robustness: The approach is robust to the choice of prior, provided there is no prior-data conflict. The inclusion of conflict checking and the ability to modify the prior ensures that the inferences are driven by the data rather than subjective prior choices.

In conclusion, the authors posit that the plausible region, derived from relative belief, provides a superior summary of evidence for particle physics problems (and general statistical inference) by ensuring that reported intervals are consistent with the likelihood function and that their reliability is quantified and controlled during the experimental design phase.