The Well-Tempered Likelihood: Honest Confidence Intervals for Misspecified Models

Technical Summary: The Well-Tempered Likelihood

Problem Statement
In high-energy physics and the physical sciences, likelihood-based inference relies on the assumption that the generative model accurately describes the observed data. However, models are frequently misspecified due to imperfect simulations, missing higher-order corrections, or unmodeled detector effects. Under misspecification, standard maximum likelihood estimation (MLE) converges to a "pseudo-true" value that minimizes the Kullback–Leibler divergence rather than the true physical parameter. Crucially, standard confidence intervals derived from Wilks' theorem shrink as $O(1/\sqrt{N})$ with increasing data size $N$. When the model is misspecified, this leads to "over-confident" constraints: intervals that become arbitrarily narrow and exclude the true value with high probability, despite the model's inadequacy. Current practices often attempt to mitigate this by adding systematic uncertainties with no statistical origin (SUnSOs) or inflating errors ad hoc, which lacks a rigorous statistical foundation.

Methodology
The authors propose the well-tempered likelihood, a modification to the standard likelihood-ratio test statistic designed to produce "honest" confidence intervals that self-limit when a model is misspecified.

The core construction defines a new test statistic $T(\theta)$:
$$T(\theta) = \frac{\Lambda(\theta)}{\hat{\gamma}}$$
where:

• $\Lambda(\theta)$ is the standard log-likelihood ratio test statistic.
• $\hat{\gamma}$ is a goodness-of-fit (GOF) statistic evaluated at the best-fit point $\hat{\theta}_{\text{MLE}}$, normalized by degrees of freedom.

The method relies on the scaling behavior of $\hat{\gamma}$:

1. Correct Specification: If the model is correct, $\hat{\gamma} \to O(1)$. The statistic $T(\theta)$ recovers the standard likelihood-ratio behavior, preserving statistical power.
2. Misspecification: If the model is incorrect, both $\Lambda(\theta)$ and $\hat{\gamma}$ scale as $O(N)$. Consequently, their ratio $T(\theta)$ remains $O(1)$. This prevents the confidence interval from collapsing to zero width. Instead, the interval stabilizes at a floor determined by the model's inadequacy.

The paper presents two specific formulations for $\hat{\gamma}$:

• Binned Case: Uses the Pearson $\chi^2$ statistic. Here, $\hat{\gamma} = \max(1, G_{\text{bin}}/ \text{dof})$. Under misspecification, the effective sample size $N_{\text{eff}} = N/\hat{\gamma}$ saturates at a constant value determined by the binning resolution and the severity of the misspecification.
• Unbinned Case: Uses a classifier-based two-sample test. A binary classifier distinguishes observed data from synthetic samples generated by the fitted model. The GOF statistic $G_{\text{cls}}$ is the deviance relative to the null hypothesis (indistinguishability). To handle the lack of a closed-form null distribution for classifiers, the authors calibrate the mean and standard deviation of $G_{\text{cls}}$ via permutation tests, converting the statistic into a shifted $z$-score for $\hat{\gamma}$.

Key Contributions

1. Self-Limiting Confidence Intervals: The primary contribution is a mechanism where confidence intervals naturally widen (or stop narrowing) when the model fails to fit the data, rather than shrinking to zero. This effectively reduces the "effective sample size" ($N_{\text{eff}}$) to a value commensurate with the model's resolution.
2. Data-Driven Uncertainty: Unlike SUnSOs, which require manual inflation of errors, the well-tempered approach derives the necessary penalty directly from the data-model discrepancy via the GOF test.
3. Dual Formulations: The paper provides both a binned formulation (analytically tractable, following an $F$-distribution under correct specification) and an unbinned formulation (suitable for high-dimensional, multivariate data using machine learning classifiers).
4. Theoretical Guarantees: The method ensures fidelity (recovering standard inference when the model is correct) and self-limitation (stabilizing intervals when the model is wrong).

Results
The authors validate the method through two case studies:

• Gaussian Proof of Concept: In a univariate Gaussian example with fixed variance, the standard interval shrinks as $1/\sqrt{N}$ even when the variance is misspecified, leading to undercoverage. The well-tempered interval tracks the standard result for small $N$ but saturates to a constant width as $N$ increases, correctly reflecting the model's inability to resolve the parameter beyond the GOF limit. The effective sample size $N_{\text{eff}}$ plateaus at a value inversely proportional to the squared deviation of the true variance from the model.
• Strong Coupling Constant ($\alpha_s$) Inference: Using synthetic $e^+e^-$ collision data and jet substructure observables, the authors employ a Deep Neural Network for the likelihood ratio (DCTR) and a gradient-boosted tree for the GOF test.
  • In the correctly specified scenario, the well-tempered and standard intervals are nearly identical.
  • In the misspecified scenario (where nuisance parameters in the simulation differ from the inference model), the standard interval becomes spuriously precise and biased. The well-tempered interval self-limits, maintaining a width that encompasses the true value, while $N_{\text{eff}}$ grows slowly and saturates (e.g., reaching $\approx 480$ effective events at $N=25,000$).

Significance and Claims
The paper claims that the well-tempered likelihood provides a principled, data-driven safeguard against overconfident constraints in the presence of model misspecification. It argues that "all models are correct, as long as your dataset is small enough," and this method formalizes that intuition by effectively reducing the dataset size until the model is adequate.

The authors emphasize that this is not a replacement for improving models but a necessary modification to inference pipelines when models are known to be imperfect approximations. The method is agnostic to the source of misspecification (e.g., parton showering, detector effects) and is particularly suited for modern, high-dimensional analyses where traditional binning is impractical. The paper concludes that while the method trades precision for honesty (resulting in wider intervals or overcoverage), this conservatism is a feature that prevents misleading physical conclusions. Future work is suggested to explore profiling nuisance parameters within the GOF test and developing GOF tests specifically optimized for parameter estimation tasks.