📊 statistics

When can a posterior predictive check identify the learning rate? Exact degeneracy in Gaussian models and implications for Generalised Bayesian Inerence

This paper demonstrates that in Gaussian linear models with a flat or reference prior, the posterior predictive check (PPC) selector for the Generalised Bayesian learning rate fails to identify the optimal value because the resulting $p$ -value becomes independent of the observed data, rendering the selection process deterministic and prone to over-tempersing.

Original authors: Nam Anh Le

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nam Anh Le

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to perfect a soup recipe. You have a standard recipe (your statistical model), but you suspect the ingredients you bought might be slightly off (model misspecification). To fix this, you decide to adjust the "heat" of your cooking process. In the world of statistics, this "heat" is called the learning rate (denoted as $\eta$ ).

If the heat is too high, the soup burns (the model overreacts to bad data). If it's too low, the flavors don't blend (the model ignores the data). The goal is to find the perfect heat setting.

Recently, a method was proposed to find this perfect heat: The Posterior Predictive Check (PPC). Think of this as a taste-tester. You simulate a new batch of soup based on your current heat setting and ask, "Does this taste like the soup I just made?" If the answer is "No" (a low score), you lower the heat and try again. You keep lowering the heat until the taste-tester says, "Okay, this is acceptable."

The Big Discovery
This paper, by Nam Anh Le, asks a very specific question: Does this taste-tester actually work when the soup is a simple, classic Gaussian (bell-curve) recipe?

The answer is a resounding no. In fact, the paper reveals that for these specific types of models, the taste-tester is "blind." It cannot tell the difference between high heat and low heat.

Here is how the paper breaks this down, using simple analogies:

1. The "Flat" Taste-Test (Known Ingredients)

Imagine you know exactly how much salt is in your ingredients (known variance).

The Paper's Claim: The taste-tester's score is completely flat. No matter what heat setting you choose, the score stays exactly the same.
The Analogy: It's like trying to judge the temperature of an oven by looking at a picture of a thermometer that is stuck on one number. The picture doesn't change whether the oven is cold or hot.
The Result: Because the score never changes, the method has no way to pick a "best" heat. It just defaults to the lowest possible setting on your dial. This makes the soup too bland (over-tempers the model), leading to predictions that are too wide and uncertain.

2. The "Ghost" Taste-Test (Unknown Ingredients)

Now, imagine you don't know how much salt is in the ingredients (unknown variance), and you use a standard, neutral way of guessing (the "reference prior").

The Paper's Claim: This is even stranger. The taste-tester's score doesn't just stay flat; it becomes independent of the data entirely.
The Analogy: Imagine a magic 8-ball that tells you the weather. Usually, it looks at the sky (the data) to give an answer. But in this specific case, the 8-ball ignores the sky completely. It gives you the exact same answer whether it's raining, sunny, or snowing.
The Result: The "best" heat setting is decided before you even cook the soup. It is a fixed number based only on how many people you are feeding ( $n$ ) and how many ingredients you have ( $d$ ). The actual taste of the soup (the data) is irrelevant.

3. The Consequence: A Broken Compass

Because the method ignores the data in these scenarios, it acts like a broken compass that always points North, regardless of where you actually are.

The Outcome: The method almost always picks the smallest, safest heat setting available.
The Cost: This results in "predictive intervals" (your estimate of how good the soup will be) that are too wide. You end up saying, "The soup might be anywhere from bland to burnt," when you could have been more precise. The paper shows this leads to over-cautious predictions that are less useful than simply using the standard recipe or testing the soup on a separate batch (held-out data).

4. Why Does This Happen? (The "Pivotal" Magic)

The paper explains that this happens because of a mathematical "trick" called pivotality.

In these specific Gaussian models with neutral priors, the math is so symmetrical that the "error" in the model cancels out the "error" in the data perfectly.
It's like a scale where the weight of the object and the weight of the counterbalance are linked in such a way that the scale always reads "balanced," no matter what you put on it. The specific structure of the Gaussian model and the "neutral" prior create this perfect cancellation.

5. The Practical Takeaway: A "Pre-Flight" Check

The paper doesn't just say "this is broken"; it offers a simple tool to avoid the problem.

The Diagnostic: Before you even run your complex model, you can do a quick, cheap calculation (a "pre-screening diagnostic").
How it works: You check if your model is close to this "Gaussian/neutral" setup. If it is, you know immediately that the taste-tester (PPC) will be blind.
The Advice: If the diagnostic says "blind," do not use the taste-tester. Instead, use a different method (like testing on a separate batch of data) to find your heat setting.

Summary

The paper reveals a hidden flaw in a popular method for tuning statistical models. For a very common class of models (Gaussian linear models with neutral priors), the method used to find the "learning rate" is data-free. It ignores the actual data and picks a default setting, leading to overly cautious and less accurate predictions. The author provides a simple way to spot this trap before you fall into it.

Technical Summary: Exact Degeneracy in Gaussian Models and Implications for Generalised Bayesian Inference

Problem Statement
Generalised (or Gibbs) Bayesian inference addresses model misspecification by tempering the likelihood function with a learning rate $\eta > 0$ , replacing $p(y|\theta)$ with $p(y|\theta)^\eta$ . The choice of $\eta$ is critical for calibration, yet selecting it remains challenging. Zafar and Nicholls (2024) proposed a simple, exploratory method for selecting $\eta$ : using a posterior predictive check (PPC) based on the log-likelihood. Specifically, one chooses the smallest $\eta$ on a grid such that the resulting PPC $p$ -value is not rejected (i.e., $p(\eta) \ge \alpha$ ). While this approach showed promise in complex, misspecified models, its behavior on simple, canonical models had not been rigorously examined. This paper investigates whether this PPC selector can distinguish between different learning rates in the canonical Gaussian linear model.

Methodology
The author conducts an exact, finite-sample analysis of the PPC $p$ -value selector on the Gaussian linear model $y = X\beta + \varepsilon$ , where $\varepsilon \sim N(0, \sigma^2 I_n)$ . The analysis considers two primary scenarios:

Known Variance: $\sigma^2 = \sigma_0^2$ is known, with a flat prior on $\beta$ .
Unknown Variance: Both $\beta$ and $\sigma^2$ are unknown, utilizing the standard reference prior $\pi(\beta, \sigma^2) \propto \sigma^{-2}$ .

The PPC $p$ -value is defined as $p(\eta) = P(T(y_{rep}, \theta) < T(y, \hat{\theta}_\eta))$ , where $T$ is the log-likelihood discrepancy, $\hat{\theta}_\eta$ is the posterior mean under the tempered likelihood, and $y_{rep}$ is drawn from the untempered model. The author derives closed-form expressions for the distribution of this $p$ -value to determine its dependence on $\eta$ , the data, and the data-generating process (DGP).

Key Contributions and Results

Exact Degeneracy under Known Variance (Theorem 1):
When $\sigma^2$ is known and a flat prior is used, the PPC $p$ -value is invariant with respect to $\eta$ . For any $\eta > 0$ , the $p$ -value equals $P(\chi^2_n > \text{RSS}/\sigma_0^2)$ .
- Implication: The selector cannot distinguish between learning rates. It either accepts the entire grid (returning the minimum grid value) or rejects the entire grid, depending solely on the residual sum of squares (RSS) relative to the assumed variance.
- Misspecification: The selector is two-sided non-identifying. If the true variance is larger than the assumed variance (over-dispersion), $p \to 0$ for all $\eta$ . If the true variance is smaller (under-dispersion), $p \to 1$ for all $\eta$ . In neither case does the selector utilize $\eta$ to correct the fit.
Data-Free Law under Unknown Variance (Theorem 2):
When $\sigma^2$ is unknown with the reference prior, the $p$ -value admits a closed-form expression:
$p(\eta) = P\left[ W + n \log\left(\frac{\eta n - d - 2}{V}\right) > \frac{\eta n - d - 2}{\eta} \right]$
where $W \sim \chi^2_n$ and $V \sim \chi^2_{\eta n - d}$ are independent.
- Crucial Finding: This expression depends only on the sample size $n$ , the rank $d$ , and the learning rate $\eta$ . It is completely independent of the realized data $y$ and the data-generating process.
- Consequence: The selected $\eta$ is determined before any data is observed. Numerical analysis shows that for typical grids, the curve $p_{n,d}(\eta)$ lies well above the rejection threshold (e.g., $\alpha=0.10$ ). Consequently, the selector collapses to the smallest admissible grid value (the "grid floor"), regardless of the data's actual fit.
Calibration Costs:
Because the selector defaults to the smallest $\eta$ (over-tempering) due to this data-free collapse, the resulting predictive intervals are excessively wide. Simulations demonstrate that 90% predictive intervals constructed with the PPC-selected $\eta$ achieve coverage rates of 0.97–0.99 (over-covering) and are 1.5–1.9 times wider than intervals derived from held-out selection or $\eta=1$ .
Role of Priors:
The exact data-independence is specific to the reference prior (or flat prior in the known variance case). The paper notes that introducing an informative prior (e.g., a proper ridge prior) breaks this exact degeneracy, making the $p$ -value dependent on the data. However, for diffuse priors, the dependence remains numerically negligible, and the selector still tends to collapse to the grid floor.

Significance and Scope
The paper positions these findings as a "boundary result" rather than a general negative statement about PPC-based learning rate selection. The significance lies in:

Identifying a Canonical Class: It delineates a specific class of models (Gaussian scale–location family with reference priors) where the PPC selector is provably non-identifying.
Mechanism Explanation: It attributes the failure to a "pivotality property" where the discrepancy statistics are functions of pivotal quantities, rendering their comparison probability free of unknown parameters and, in the unknown variance case, free of the data itself.
Practical Diagnostic: The author proposes a "cheap, data-free pre-screening diagnostic." Before fitting a model, practitioners can compute the theoretical $p$ -value curve for their $(n, d)$ configuration. If the minimum $p$ -value on the grid exceeds the significance level $\alpha$ , the PPC selector is known in advance to be data-free and will likely collapse to the grid floor. In such regimes, data-driven alternatives (e.g., held-out selection or ELPPD) are recommended.

The paper concludes that while the Zafar and Nicholls (2024) approach may work in complex, non-conjugate settings, it fails fundamentally in the canonical Gaussian setting due to these exact finite-sample properties, necessitating caution when applying it to models with similar structural characteristics.