📊 statistics

Approximate posterior recalibration

This paper introduces two methods that extend simulation-based calibration (SBC) to recalibrate approximate Bayesian posteriors by widening their uncertainty intervals to achieve marginal calibration, addressing the common issue of overly narrow interval estimates.

Original authors: Tiffany Cai, Philip Greengard, Ben Goodrich, Andrew Gelman

Published 2026-03-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Tiffany Cai, Philip Greengard, Ben Goodrich, Andrew Gelman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a weather forecaster. You have a sophisticated computer model that predicts the temperature for tomorrow. Usually, your model is great, but sometimes it gets a little too confident. It says, "It will be 70 degrees," with a tiny margin of error, like "70 ± 0.1 degrees."

In reality, the temperature might be 68 or 72. Your model is miscalibrated: it thinks it knows more than it actually does. If you use this model to decide whether to wear a coat, you might get cold because the model didn't account for enough uncertainty.

This paper is about fixing that overconfidence in complex mathematical models used by scientists and data analysts.

The Problem: The "Too Narrow" Net

In statistics, when we try to guess a hidden number (like the true effect of a drug), we use a "posterior distribution." Think of this as a net we cast to catch the true answer.

Perfect Model: The net is the perfect size. If we cast it 100 times, it catches the true fish 95 times (if we aimed for 95% confidence).
Approximate Model: Because the math is too hard to solve exactly, we use a shortcut (an approximation). These shortcuts often cast a net that is too narrow. They catch the fish less often than they should, making us falsely confident.

The authors ask: How do we widen this net just enough so it catches the fish the right amount of the time, without making it so wide it catches everything?

The Solution: The "Practice Run" (Simulation-Based Calibration)

To fix the net, you need to know how it behaves. The authors use a clever trick called Simulation-Based Calibration (SBC).

Imagine you are a fisherman who wants to test your net. You can't just wait for real fish. Instead, you:

Fake the Ocean: You generate 1,000 fake scenarios where you know exactly where the fish should be (because you made them up).
Cast the Net: You run your approximate model on these fake scenarios.
Check the Score: You see how often your net actually caught the fish you planted.

If your net is too narrow, you'll see that it misses the fish way too often. The paper shows that if you do this enough times, you can see a pattern: "Oh, my net is consistently 20% too small."

The Fix: Two Ways to Widen the Net

Once you know the net is too small, you need to stretch it. The paper proposes two simple ways to figure out how much to stretch it:

The "Target Score" Method (Nominal Coverage):
You say, "I want my net to catch the fish 95% of the time." You try stretching the net by 10%, then 20%, then 30%, testing each one on your fake ocean until you find the exact stretch that hits the 95% target.
- Pros: Very precise for specific goals.
- Cons: Takes a lot of computer time to test every stretch.
The "Z-Score" Method:
Instead of aiming for a specific percentage, you look at the average "miss" distance. You calculate how far off your net usually is and stretch it just enough to make the average error look normal.
- Pros: Much faster and easier to calculate.
- Cons: It's a "one-size-fits-all" stretch, which might not be perfect for every single scenario, but it works well enough.

The authors tested these methods on simple math problems and a famous "8 schools" dataset (a standard test case in statistics). They found that by applying these stretches, their "too narrow" nets became perfectly sized again.

The Twist: The "Posterior Recalibration" Paradox

Here is where it gets tricky and philosophical.

Usually, you test your net by simulating the whole ocean (the Prior). This is the standard, correct way to do things in Bayesian statistics.

But what if you only care about the part of the ocean where the fish actually are? What if you say, "I don't care about the deep, dark parts of the ocean where no fish live; I only want to test my net in the shallow, sunny spots where my data says the fish are"?

This is called Posterior Recalibration. You use your current best guess to generate the fake scenarios.

The Catch:
The paper discovered a surprising flaw in this approach. If you test your net only in the "sunny spots" (the posterior), the math breaks down.

The Analogy: Imagine you are calibrating a speedometer. If you only test it while driving at 60 mph, and you adjust it to read 60 mph perfectly, you might break the calibration for driving at 20 mph or 80 mph.
The Result: When the authors tried this "Posterior Recalibration," the model became overconfident again, but in a different way. It stopped "pooling" information correctly. It became too sure of itself, ignoring the wisdom of the broader data.

Why do it then?
The authors suggest that in very complex, multi-layered models (like hierarchical models with many groups), this "flawed" method might actually be a useful compromise. It forces the model to focus on the specific data at hand rather than getting lost in theoretical possibilities, even if it technically breaks the rules of pure Bayesian math.

The Big Picture

The Goal: Make sure our statistical "nets" aren't too tight, so we don't get falsely confident.
The Tool: Run thousands of fake simulations to see how the model performs, then stretch the results to fix the errors.
The Warning: There is a "right way" (testing against the whole universe of possibilities) and a "pragmatic way" (testing only against what we see). The pragmatic way is tempting but mathematically risky, though it might work in specific, complex situations.

In short, this paper gives us a toolkit to audit our own confidence. It teaches us how to take a model that is "almost right" but "too sure of itself," and gently nudge it back to a healthy, realistic level of uncertainty.

1. Problem Statement

Bayesian inference often relies on approximate algorithms (e.g., Variational Inference, ADVI) when exact posterior computation is computationally intractable. While exact Bayesian posteriors are theoretically "calibrated" (meaning posterior intervals contain the true parameter value with the nominal frequency when averaged over the prior predictive distribution), approximate posteriors frequently suffer from miscalibration. Specifically, they often produce uncertainty intervals that are too narrow, failing to capture the true estimation error.

The core challenge addressed is: How can we adjust approximate posterior distributions to achieve proper uncertainty quantification (calibration) without resorting to computationally expensive exact methods?

The paper distinguishes between two types of calibration:

Prior Calibration: Averaging over the prior distribution (the standard Bayesian definition).
Posterior Calibration: Averaging over the posterior distribution (focusing on the region of parameter space supported by the observed data).

2. Methodology

The authors propose a framework called Approximate Posterior Calibration (APC), which extends Simulation-Based Calibration (SBC).

2.1. The APC Framework (Algorithm 1)

The general procedure involves three steps:

Simulation: Draw $L$ parameters $\Theta_l$ from the prior $p(\Theta)$ , generate synthetic data $y_l$ from the likelihood $p(y|\Theta_l)$ , and fit the approximate model $M'$ to obtain approximate posterior draws $\theta_{l,s}^{post}$ .
Adjustment Learning: Calculate an adjustment factor (e.g., a scaling factor $k$ ) that forces the approximate posterior to satisfy a specific self-consistency property derived from the exact posterior theory.
Application: Apply the learned adjustment to the approximate posterior of the actual observed data $y_D$ .

2.2. Proposed Adjustment Methods

The authors focus on a class of adjustments involving fixed-width scaling around the posterior mean. Two specific methods for determining the scaling factor are proposed:

A. Nominal Coverage Method:
- Goal: Ensure that $(1-\alpha)$ credible intervals achieve nominal coverage (i.e., the true parameter falls within the interval $1-\alpha$ of the time).
- Mechanism: Perform a grid search over scaling factors $\sigma$ to minimize the squared difference between the observed quantile coverage and the target $(1-\alpha)$ .
- Pros/Cons: Directly targets interval coverage but is computationally expensive due to the grid search and can be unstable for extreme $\alpha$ values.
B. Z-Score Method:
- Goal: Ensure that the standardized residuals (z-scores) of the true parameter relative to the approximate posterior follow a standard normal distribution $N(0,1)$ .
- Mechanism: Calculate the z-scores $z_l = (\theta_l - \mu_l^{post}) / \sigma_l^{post}$ across simulations. The scaling factor is set to the standard deviation of these z-scores ( $s_z$ ).
- Pros/Cons: Computationally efficient (no grid search) and stable, though it assumes a single scaling factor applies to all interval widths.

2.3. Posterior Recalibration (Alternative Approach)

The paper explores a variant where the simulation step draws $\Theta_l$ from the posterior $p(\theta|y)$ rather than the prior.

Motivation: Focuses recalibration on the region of parameter space actually supported by the data, ignoring prior regions inconsistent with observations.
Theoretical Finding: The authors prove analytically (using a Normal-Normal model) that posterior recalibration is fundamentally flawed for non-hierarchical models. Even with exact inference, averaging over the posterior does not yield uniform quantiles. Applying recalibration in this context systematically reduces "partial pooling" (pulling estimates toward the prior) and results in overconfident (too narrow) intervals.

3. Key Contributions

Formalization of APC: The paper introduces a systematic procedure to widen approximate posterior intervals to achieve marginal calibration with respect to the prior.
Two Practical Algorithms: It proposes and compares the Nominal Coverage and Z-Score methods for determining the necessary scaling factor.
Theoretical Analysis of Posterior Recalibration: It provides a rigorous derivation showing that while prior-based recalibration works, posterior-based recalibration (averaging over the posterior) generally fails to produce calibrated intervals for simple models, leading to under-coverage and reduced pooling.
Hierarchical Model Insight: The authors hypothesize that posterior recalibration might still be viable for hierarchical models due to "internal replication" (where new group parameters act like prior draws), though this requires further investigation.

4. Experimental Results

The methods were tested on two primary settings:

Simple Gaussian Model (One Parameter):
- Setup: A known model where the true posterior is available. The approximate posterior was artificially narrowed (scaled by 1/3).
- Result: Both APC methods successfully recovered the true scaling factor (approx. 3.0).
- Performance: The Z-Score method was more efficient (no grid search) and yielded scaling factors ( $\sigma \approx 3.08$ ) very close to the ground truth, restoring nominal coverage (e.g., 95% coverage became 95.1%).
Hierarchical 8-Schools Model (ADVI vs. HMC):
- Setup: Used Automatic Differentiated Variational Inference (ADVI) as the approximate method and Hamiltonian Monte Carlo (HMC) as the ground truth.
- Result: ADVI posteriors were uncalibrated. The APC methods identified a scaling factor of roughly 2.4–2.5.
- Nuance: The authors noted that ADVI posteriors were not just narrower but also shifted relative to HMC. Since APC only corrects width (not shift), the adjusted intervals were wider than a simple 2x scaling of the ADVI width, ensuring the true parameter was captured. The adjusted intervals achieved near-nominal coverage (e.g., 95% target $\to$ 94.9% observed).

5. Significance and Discussion

Practical Utility: The APC framework offers a "plug-in" solution for practitioners using approximate inference (like VI) to ensure their uncertainty estimates are reliable. It transforms a potentially misleading narrow interval into a calibrated one.
Theoretical Boundaries: The paper clarifies a critical distinction in Bayesian workflow. While prior predictive calibration is a natural property of Bayesian inference, posterior predictive calibration is not. Attempting to force calibration over the posterior distribution in simple models leads to biased inferences (less pooling, overconfidence).
Future Directions: The authors suggest that while posterior recalibration is theoretically problematic for simple models, it may serve as a robustness tool or be effective in hierarchical models where the internal structure mimics prior predictive checks. They also highlight that current methods focus on scalar parameters; extending this to joint multivariate calibration remains an open challenge.

In summary, the paper provides a practical, simulation-based toolkit to fix the under-coverage of approximate Bayesian posteriors, while simultaneously warning against the theoretical pitfalls of trying to recalibrate based on the posterior distribution itself in non-hierarchical settings.