📊 statistics

Efficient Uncertainty Propagation in Bayesian Two-Step Procedures

This paper proposes an efficient two-step Bayesian inference framework that utilizes Pareto smoothed importance sampling and iterative moment matching to approximate posterior distributions via mixture models, thereby significantly reducing the computational cost of propagating uncertainty in surrogate modeling and missing data imputation without sacrificing accuracy.

Original authors: Svenja Jedhoff, Hadi Kutabi, Anne Meyer, Paul-Christian Bürkner

Published 2026-08-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Svenja Jedhoff, Hadi Kutabi, Anne Meyer, Paul-Christian Bürkner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of scientific modeling, researchers often face a dilemma: the most accurate way to understand a complex system is to run a massive, detailed simulation, but these simulations can take days or even weeks to complete on a single computer. To speed things up, scientists frequently use a "two-step" approach. First, they run the expensive simulation a limited number of times to build a simpler, faster approximation, often called a surrogate model. Second, they use this fast approximation to make predictions or draw conclusions about the real world. However, this shortcut introduces a hidden problem. Because the fast model is built on limited data, it carries its own uncertainty, and the real-world data used in the second step might be incomplete or missing pieces. If a researcher ignores these uncertainties and treats the fast model as perfect, their final conclusions can be dangerously overconfident, leading to errors in everything from climate predictions to medical diagnoses. The challenge has been to carry the uncertainty from the first step all the way through to the final answer without having to run the expensive simulation thousands of times, which would defeat the purpose of using a shortcut in the first place.

A team of statisticians has developed a new method to solve this specific bottleneck, allowing researchers to propagate uncertainty efficiently without sacrificing accuracy. Their approach is designed for situations where a scientist must account for many different possibilities—such as multiple versions of a dataset where missing values have been filled in, or many different versions of a fast approximation model. Traditionally, to get a reliable answer, a researcher would have to run the heavy, slow simulation separately for every single one of these possibilities. If there were one hundred different scenarios to test, they would have to run the computer program one hundred times, a process that consumes vast amounts of time and energy. The new method drastically cuts this workload by realizing that most of these scenarios are similar enough that they do not need a fresh, full simulation. Instead, the researchers run the expensive simulation just a few times to get a solid reference point. They then use a clever statistical technique to estimate what the results would look like for the remaining scenarios, effectively "borrowing" information from the few expensive runs to fill in the gaps for the many others.

The core of this technique involves a safety check that ensures the estimates remain trustworthy. When the researchers try to estimate the results for a new scenario based on an old one, they first check how similar the two situations really are. If the situations are close, the estimate is accepted immediately. If the situations are too different, the method automatically switches to a more sophisticated adjustment process to correct the estimate before accepting it. This prevents the method from making wild guesses when the data changes significantly. The researchers tested this approach in two very different real-world contexts. In the first, they looked at a common problem where data is missing, such as a survey where some people forgot to answer certain questions. They generated one hundred different versions of the survey to account for the missing answers and then tried to predict travel times for New York City taxis. In the second test, they simulated a complex physical system using a fast approximation model to infer unknown parameters.

The results showed that the new method could achieve the same level of accuracy as the traditional, brute-force approach while using a fraction of the computing power. In the taxi travel time study, the new method required only about five percent of the computational effort needed by the standard method. In the simulation studies, the number of times the researchers had to run the expensive computer program dropped from one hundred to as few as one or two in many cases, depending on how complex the data was. The method proved robust even when the data was messy or the missing information was substantial, though it did face challenges when the underlying mathematical models were extremely complex and the missing data was very high. By combining these efficient estimation techniques with a rigorous diagnostic check, the researchers have provided a tool that allows scientists to handle uncertainty properly without being held back by the limits of their computer hardware. This means that in fields ranging from engineering to public health, researchers can now build more reliable models that account for all sources of error, ensuring that their final conclusions are as trustworthy as they are fast.

Technical Summary: Efficient Uncertainty Propagation in Bayesian Two-Step Procedures

Problem Definition

Bayesian inference often employs a two-step framework where an initial modeling or preprocessing phase (generating uncertain quantities $\tau$ ) is followed by a subsequent inference or prediction phase (estimating parameters $\theta$ ). Common applications include surrogate modeling, where a complex simulator is approximated by a simpler model, and missing data problems, where multiple imputation (e.g., MICE) generates several completed datasets.

The core challenge in these workflows is uncertainty propagation. To obtain the marginalized posterior $p(\theta | C)$ , one must integrate over the uncertainty in the first step:
$p(\theta | C) = \int p(\theta | \tau, C)p(\tau | C) d\tau$
Standard approaches approximate this integral via Monte Carlo estimation by fitting a separate, computationally expensive Bayesian model (typically using MCMC) for every realization $\tau^{(i)}$ of the uncertain quantity. If $m$ realizations are generated, this requires $m$ separate MCMC runs. This "brute-force" approach becomes computationally prohibitive as $m$ increases or as the models become complex, particularly in settings involving high-dimensional parameters or intricate likelihoods.

Methodology

The authors propose an efficient iterative algorithm that reduces computational overhead by leveraging importance sampling techniques, specifically Pareto Smoothed Importance Sampling (PSIS) and Importance Weighted Moment Matching (IWMM).

Core Mechanism

Instead of running MCMC for every $\tau^{(i)}$ , the method selects a small subset of "representative" values (or a mixture of them) to serve as a proposal distribution $p(\theta | \tau^*)$ . MCMC is run only for these representatives. The remaining target posteriors $p(\theta | \tau^{(i)})$ are approximated by reweighting the samples from the proposal distribution.

PSIS: The primary approximation method. It uses importance weights to transform samples from the proposal $p(\theta | \tau^*)$ to approximate the target $p(\theta | \tau^{(i)})$ .
- Diagnostic: The method employs the Pareto shape parameter $\hat{k}$ to assess reliability. If $\hat{k} < 0.7$ , the approximation is deemed reliable.
- Smoothing: PSIS smooths the tail of the importance weights using a generalized Pareto distribution to stabilize variance.
IWMM (Adaptive Correction): If PSIS fails (i.e., $\hat{k} \geq 0.7$ ), indicating poor overlap between proposal and target, the method applies IWMM.
- This adaptive technique iteratively updates the proposal distribution via affine transformations (mean shift, variance matching, or full covariance matching) to better align with the target moments.
- The process continues until the diagnostic $\hat{k}_{MM}$ falls below the threshold or the most complex transformation fails to improve the fit.
Iterative Algorithm:
- The algorithm maintains a set of unapproximated targets.
- In each iteration, it selects a representative $\tau^*$ (from the remaining set or a mixture thereof), runs MCMC to obtain samples, and attempts to approximate all remaining targets using PSIS.
- Targets where PSIS fails are handled by IWMM.
- If IWMM also fails, a new representative is selected in the next iteration.
- This loop continues until all $m$ posteriors are approximated, after which the marginalized posterior is computed via a Monte Carlo average.

Specialized Implementations

Missing Data: The method utilizes the structural similarity of imputed datasets (which differ only in missing entries) to optimize the calculation of importance ratios, evaluating likelihoods only for the differing rows. Representative datasets are selected using clustering (k-medoids) in the data space.
Surrogate Modeling: Since the similarity of surrogate parameters $\tau$ does not guarantee similarity of the resulting posteriors, the authors propose selecting representatives based on the average log-likelihood of the observed data evaluated over prior samples of $\theta$ .

Key Contributions

Efficient Uncertainty Propagation: The paper introduces a framework that significantly reduces the number of required MCMC runs (and associated log-probability/gradient evaluations) while maintaining the integrity of uncertainty propagation.
Robustness via Diagnostics: By integrating the PSIS diagnostic $\hat{k}$ and the adaptive IWMM correction, the method automatically detects and addresses cases where importance sampling is unreliable, preventing the propagation of biased estimates.
Mixture Proposals: The authors demonstrate that using a mixture of multiple representative posteriors as a proposal distribution can improve coverage and efficiency, particularly when a single representative is insufficient to capture the diversity of the posterior space.
Domain-Specific Optimizations: The paper provides tailored strategies for selecting representatives in both missing data (clustering in data space) and surrogate modeling (log-likelihood ranking) contexts.

Results

The method was evaluated through simulation studies and a real-world case study:

Multiple Imputation Simulations:
- In scenarios with $N=100$ observations and varying missingness (5%–30%), the proposed PSIS+IWMM approach required substantially fewer log-probability and gradient evaluations compared to the brute-force MCMC baseline.
- For lower-dimensional problems ( $p=10$ ), the method reduced computational cost by more than 50%.
- In high-dimensional settings ( $p=30$ ) with complex priors (regularized horseshoe) and high missingness (30%), the method struggled to reduce MCMC runs effectively, sometimes exceeding the cost of the baseline, though it still provided accurate approximations.
- Posterior summaries (mean, standard deviation, quantiles) from the proposed method showed negligible absolute differences ( $<0.025$ ) compared to the MCMC baseline.
- Kernel Maximum Mean Discrepancy (MMD) tests confirmed that the approximated distributions were statistically indistinguishable from the baseline in most cases.
NYC Taxi Case Study:
- Applied to a travel time prediction task with missing predictor data.
- The PSIS+IWMM method required only ~8% of the log-probability evaluations and ~5% of the gradient evaluations compared to the brute-force approach.
- The resulting marginalized posterior distributions closely matched the gold-standard HMC results.
Surrogate Modeling Simulations:
- Tested on logistic and Polynomial Chaos Expansion (PCE) surrogates.
- The iterative algorithm successfully approximated the marginalized posterior of the input parameters, demonstrating effectiveness in propagating surrogate model uncertainty.

Significance and Claims

The paper claims to offer a practical and scalable solution for Bayesian two-step workflows where uncertainty propagation is critical but computationally expensive. The authors emphasize that their method:

Preserves Accuracy: It maintains the reliability of uncertainty quantification by strictly adhering to diagnostic thresholds ( $\hat{k} < 0.7$ ) and only accepting approximations that pass these checks.
Reduces Computational Burden: It significantly lowers the cost of inference in scenarios requiring multiple model fits, making complex Bayesian two-step procedures feasible for larger datasets and more models.
Is Robust: The inclusion of IWMM ensures that the method does not fail silently when the proposal distribution is a poor match for the target, a common failure mode in standard importance sampling.

The authors remain modest regarding limitations, noting that in highly complex, high-dimensional settings with strong shrinkage priors and high missingness, the efficiency gains may diminish, and the method may not always outperform brute-force MCMC in terms of raw computational cost, though it still provides a valid approximation framework.