📊 statistics

Gaussian Mean Field Variational Inference can Overestimate Predictive Variance

This paper challenges the conventional wisdom that Mean Field Variational Inference always underestimates uncertainty by demonstrating that, in Bayesian Linear Regression, it can actually overestimate predictive variance on in-distribution data due to a trade-off where underestimation in some parameter directions necessitates overestimation in others, a phenomenon that can be mitigated through temperature scaling.

Original authors: James Odgers, Ben Riegler, Siddharth Swaroop, Vincent Fortuin

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: James Odgers, Ben Riegler, Siddharth Swaroop, Vincent Fortuin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Misunderstanding

For a long time, scientists believed that a specific method called Mean Field Variational Inference (MFVI) was always "too confident." They thought it looked at the data, made a guess, and then said, "I'm pretty sure about this," even when it should have been more uncertain. In technical terms, they thought it underestimated the variance (uncertainty).

This paper flips that story on its head. The authors show that while MFVI is indeed too confident about the internal details of the model, it can actually be too unsure (overestimate uncertainty) when it comes to making predictions on data that looks like the training data.

The Analogy: The Map and the Road

Imagine you are trying to draw a map of a city based on a few photos of a single street.

The "Internal" View (Parameter Space):
If you look at the map itself, MFVI draws a very tight, small circle around the street. It thinks, "I know exactly where this street is." It ignores the fact that the rest of the city is foggy and unknown. In this sense, it is too confident about the map's details.
The "Prediction" View (Predictive Space):
Now, imagine you are a driver trying to predict where the road goes ahead of you.
- The Exact Truth: The real road is straight and narrow. The uncertainty is low because the road is well-defined.
- The MFVI Prediction: Because MFVI drew such a tiny, tight circle around the street in step 1, it gets confused about how that tight circle translates to the road ahead. It ends up drawing a huge, fuzzy cloud of "maybe the road goes left, maybe right, maybe it disappears."
- The Result: On the actual road (the training data), MFVI is too scared. It thinks the road is much more dangerous and unpredictable than it really is.

The "Balancing Act" (The See-Saw)

The paper explains a strange rule about this method: It cannot be wrong in just one way.

Think of the model's uncertainty as a see-saw.

If the model is too confident (underestimates uncertainty) in some directions, it must be too unsure (overestimates uncertainty) in other directions to keep the math balanced.
The paper proves that for data that looks like the training data (In-Distribution), the "too unsure" side of the see-saw is the one that wins. The model overestimates the risk.

The "Cold" Fix

The authors discovered a way to fix this over-cautiousness. In the world of Bayesian math, there is a concept called Temperature.

High Temperature: Makes the model more relaxed and spread out (more uncertain).
Low Temperature (Cold): Makes the model sharper and more focused (less uncertain).

Usually, people think you need to "cool down" a model to make it less confident. But here, because the model is already too scared (overestimating uncertainty), the authors found that lowering the temperature (making it "Cold") actually helps.

It's like telling a nervous driver: "You are overestimating the danger. Take a deep breath and focus." By lowering the temperature, the model's predictions snap back to reality, matching the true, exact answer much better.

The High-Dimensional Trap

The paper shows a scary scenario where this gets worse as things get more complex (high dimensions).

Imagine trying to navigate a city with 1,000 streets instead of just one.
If all your photos are of just one specific street, the MFVI model gets so confused by the other 999 empty streets that it forgets the one street it actually saw.
It ends up predicting that the road is just as uncertain as if it had seen no data at all. It completely fails to learn from the training data.
However, applying the "Cold" fix (lowering the temperature) saves the day, allowing the model to realize, "Oh, I do know something about this street," and corrects its prediction.

The "Cold Posterior Effect"

In the world of Deep Learning, researchers have noticed that models often work better when they are "cooled down" (a phenomenon called the Cold Posterior Effect). Usually, people blamed this on bad data or bad models.

This paper offers a new reason: Even with perfect data and a perfect model, the math of MFVI naturally makes the model too scared. Therefore, "cooling" the model isn't a hack; it's a necessary correction to stop the model from overestimating uncertainty on data it should know well.

Summary

Old Belief: MFVI is always too confident.
New Discovery: MFVI is too confident about the model's internal settings, but too uncertain about the predictions it makes on familiar data.
The Fix: Lowering the "temperature" (making the posterior "Cold") corrects this over-cautiousness, making the predictions accurate again.
The Warning: In very complex, high-dimensional situations, this over-cautiousness can get so bad that the model forgets what it learned entirely, unless you apply the "Cold" fix.

Technical Summary: Gaussian Mean Field Variational Inference Can Overestimate Predictive Variance

Problem Statement
Mean Field Variational Inference (MFVI) is widely characterized in the literature as an approximation that underestimates posterior variance, leading to overconfident predictions. This characterization stems from the observation that MFVI, by factorizing the posterior across parameters, acts as a mode-fitting approximation in parameter space, avoiding low-probability regions and thus shrinking the variance of the parameter distribution. However, this paper argues that this characterization is incomplete when the objective shifts from parameter inference to prediction. Specifically, the authors demonstrate that while MFVI may underestimate variance in the parameter space, it can paradoxically overestimate the predictive variance for test points drawn from the training distribution (In-Distribution or ID data). This overestimation is particularly pronounced in directions where the training data concentrates.

Methodology
The authors conduct a rigorous theoretical analysis using Conjugate Bayesian Linear Regression (BLR) as a tractable setting to derive exact results.

Setup: They assume a linear model $y = \theta^\top x + \epsilon$ with Gaussian noise, a Gaussian prior $\theta \sim \mathcal{N}(0, \alpha^{-1}I)$ , and a training dataset $D$ .
Exact vs. Approximate: They compare the exact posterior $p(\theta|D)$ , which is a Multivariate Gaussian with covariance $\Sigma$ , against the MFVI posterior $q(\theta) = \mathcal{N}(m, S)$ , where $S$ is constrained to be diagonal.
Theoretical Derivation:
- They derive that the optimal MFVI mean $m^*$ equals the exact posterior mean $\mu$ .
- They show that the optimal MFVI covariance eigenvalues $d_p^*$ are the weighted harmonic means of the exact posterior eigenvalues $\delta_q$ , weighted by the squared projections of the exact posterior's eigenvectors onto the canonical basis vectors.
- Using these properties, they analyze the predictive variance $x^\top S^* x$ versus $x^\top \Sigma x$ for different test point distributions.
Temperature Scaling: To address the discrepancy, the authors explore the Cold Posterior Effect (CPE). They define a "Cold MFVI" posterior by minimizing the KL divergence to a tempered (cold) posterior $p_T(\theta|D) \propto p(D|\theta)^{1/T}p(\theta)^{1/T}$ with $T < 1$ . In the conjugate Gaussian case, this simply scales the MFVI covariance by $T$ ( $S^*_T = T S^*$ ).

Key Contributions

Theoretical Proof of Overestimation: The paper proves that for BLR with spherical priors, MFVI overestimates predictive variance along the first principal component of the training data. Crucially, for test points distributed according to the empirical training distribution, the expected predictive variance of MFVI exceeds that of the exact posterior (Theorem 3.7).
Calibration Constraint: The authors establish a "calibration" property (Lemma 3.6): the ratio of MFVI variance to exact variance, averaged over the eigenvectors of the true posterior, equals 1. This implies that if MFVI underestimates variance in some directions (typically those orthogonal to the data), it must overestimate it in others (typically the directions of high data variance) to compensate.
Pathological High-Dimensional Limit: In a pathological setup where data lies on a rank-1 subspace, the authors show that as the dimension $P \to \infty$ , the MFVI posterior fails to reduce predictive variance compared to the prior at all. The information from the single data direction is diluted across the $P$ axis-aligned components, causing the MFVI predictive to collapse toward the prior predictive.
Reinterpretation of the Cold Posterior Effect: The paper provides a novel explanation for the Cold Posterior Effect (CPE) in the absence of model mismatch or data augmentation. They argue that lowering the temperature ( $T < 1$ ) corrects the overestimation of predictive variance on ID data, bringing MFVI predictions closer to the exact posterior. Conversely, for Out-of-Distribution (OOD) data, "warm" posteriors ( $T > 1$ ) may be required to correct underestimation.

Results

Synthetic Experiments: In low-dimensional settings ( $P=2$ ), MFVI at $T=1$ approximates the exact posterior well. However, in high-dimensional settings ( $P=1024$ ) with rank-deficient data, MFVI at $T=1$ becomes indistinguishable from the prior, failing to capture the information in the data.
Temperature Tuning: Experiments on synthetic and real-world UCI regression tasks (using Basis Function Regression) confirm that:
- For ID test data, the optimal temperature $T^*$ is consistently less than 1, minimizing various statistical divergences (KL, Wasserstein, etc.) between the MFVI and exact predictive distributions.
- For OOD test data (orthogonal to training directions), the optimal temperature is often greater than 1.
Basis Function Regression: Applying these insights to non-linear regression via Radial Basis Functions (RBF) shows that the overestimation pathology scales with the number of basis functions (dimensionality), mirroring the linear regression findings.

Significance and Claims
The paper claims to provide a mechanistic understanding of why MFVI can be underconfident (overestimate variance) on in-distribution data, challenging the standard dogma that MFVI is always overconfident.

Completeness of Characterization: The authors assert that the "MFVI underestimates uncertainty" narrative is incomplete; the direction of the error depends on the test distribution relative to the training data.
Justification for CPE: The work offers a new justification for the Cold Posterior Effect, distinct from explanations involving misspecified likelihoods or priors. It suggests that tempering is a necessary correction for the specific approximation errors introduced by the mean-field assumption in high-dimensional, correlated data settings.
Practical Implication: The findings suggest that for tasks where test data resembles training data, applying a cold posterior (lowering temperature) to MFVI can significantly improve predictive calibration and performance, aligning the approximate posterior more closely with the exact Bayesian solution.

The authors remain modest, noting that their theoretical results are derived for conjugate linear models, but they demonstrate empirically that these intuitions extend to non-linear basis function regression and real-world datasets. They do not claim to solve all VI limitations but rather to correct a specific, counter-intuitive failure mode of MFVI in predictive settings.