Efficient prior sensitivity analysis for Bayesian model comparison

Technical Summary: Efficient Prior Sensitivity Analysis for Bayesian Model Comparison

Problem Statement
Bayesian model comparison relies on the calculation of the marginal likelihood (Bayesian evidence), which inherently embodies Occam's razor by penalizing models with excessive or wasted parameter space. However, this mechanism renders the evidence directly sensitive to the specification of the prior volume. While best practices dictate that priors should be physically motivated or data-informed, the resulting model comparison can be arbitrary if priors are not carefully chosen. Consequently, assessing the sensitivity of the evidence to plausible alternative priors is critical for robust scientific inference.

Currently, such sensitivity analyses are computationally prohibitive. Assessing the influence of alternative priors typically requires either repeated, costly model re-fits (running full Markov chain Monte Carlo or nested sampling from scratch) or specialized sampling schemes. Existing formal sensitivity analysis methods for posterior inference do not extend to Bayes factors, and "informal" checks are often only tractable in simple limiting cases. This work addresses the need for an efficient method to assess prior dependence without incurring the computational cost of re-running entire sampling procedures.

Methodology
The authors propose an approach that decouples the sampling process from the evidence calculation, enabling the reuse of existing posterior draws for prior sensitivity analysis. The method consists of two primary components:

1. Importance Resampling (SIR):
   Given a set of posterior samples $\{\theta_i\}$ drawn from an original model $M_0$ (defined by likelihood $L$ and prior $p(\theta|M_0)$), the method approximates the posterior for an alternative model $M_1$ (with the same likelihood but different prior $p(\theta|M_1)$) using Sampling Importance Resampling (SIR).

   • Unnormalized importance weights are calculated as the ratio of the new prior to the old prior: $w_i = p(\theta_i|M_1) / p(\theta_i|M_0)$.
   • A new set of unweighted samples $\{\theta^*_i\}$ is generated by resampling the original set with replacement according to these weights.
   • This step avoids any new likelihood evaluations, as the likelihood terms cancel out in the weight calculation.

2. Learned Harmonic Mean Estimator (LHME):
   To calculate the evidence for the alternative model using the resampled draws, the authors employ the Learned Harmonic Mean Estimator.

   • The LHME introduces a learned target distribution $\phi(\theta)$ (typically a normalizing flow trained on posterior samples) to stabilize the standard harmonic mean estimator.
   • The evidence for the alternative model is estimated using the resampled draws and the new prior, without re-evaluating the likelihood.
   • If the posterior is insensitive to the prior change, the original learned target $\phi_0(\theta)$ is reused. If the posterior shifts significantly, $\phi(\theta)$ is re-trained on the resampled draws (a process significantly faster than a full model re-fit).

Diagnostic Checks
To ensure the reliability of the importance sampling, the authors implement a two-stage diagnostic workflow:

• Pareto-$\hat{k}$ Diagnostic: Estimates the shape parameter of the generalized Pareto distribution fitted to the upper tail of importance weights. If $\hat{k} > 0.7$, the importance sampling is deemed unreliable, indicating that a full model re-fit is necessary.
• Effective Sample Size (ESS): Calculates the fractional ESS of the importance weights. If the ESS drops below a conservative threshold (e.g., 0.95), it indicates low overlap between the original and new posteriors, triggering the re-training of the target density $\phi(\theta)$ to maintain low variance in the evidence estimate.

Key Contributions

• Decoupling Sampling and Evidence: The work demonstrates that by leveraging the LHME, one can decouple the generation of posterior samples from the calculation of the evidence. This allows posterior draws from any sampling method (MCMC, nested sampling, neural density estimators) to be reused for alternative priors.
• Computational Efficiency: The method eliminates the need for repeated likelihood evaluations during sensitivity analysis. It provides a pathway to assess prior dependence that is agnostic to the original sampling strategy.
• Robust Diagnostics: The integration of Pareto-$\hat{k}$ and ESS diagnostics provides a systematic way to flag when importance resampling fails and when a full re-fit is unavoidable, preventing the reporting of erroneous evidence estimates.

Results
The method was validated on two toy problems and a cosmological case study:

1. Gaussian Toy Problem: In a 10-dimensional Gaussian setting, the method accurately reproduced analytical evidence values across varying prior widths. Diagnostics correctly identified when the posterior was insensitive (high ESS, no re-training needed) versus when the prior became so informative that the posterior shifted drastically (low ESS, high $\hat{k}$, requiring re-training or re-fit).
2. Rosenbrock Toy Problem: Using a non-linear Rosenbrock likelihood, the method successfully estimated evidence for informative priors shifted away from the likelihood mode. It correctly flagged the most shifted prior as unreliable via the Pareto-$\hat{k}$ diagnostic.
3. Cosmological Case Study: Applied to the tensor-to-scalar ratio ($r$) in cosmology, the method compared logarithmic and uniform priors. It reproduced evidence estimates consistent with full nested sampling re-fits and MCMC re-runs.
   • Speed-up: For the cosmological example, the resampled approach achieved computational speed-ups of approximately 3000× to 6000× compared to full model re-fits.
   • Accuracy: The estimated log-evidence values matched those from full re-fits within quoted uncertainties, even when the posterior changed moderately or significantly.

Significance and Claims
The authors claim that this approach offers a practical solution to a long-standing bottleneck in Bayesian model comparison: the high cost of prior sensitivity analysis. By enabling the direct calculation of evidence from resampled posterior draws, the method dramatically reduces computational costs while remaining agnostic to the sampling technique used to generate the original samples.

The paper emphasizes that while the fragility of Bayesian evidence to prior choice is a known issue, the inability to efficiently test alternative priors has hindered transparent reporting. This method facilitates such transparency, allowing researchers to explore the sensitivity of their conclusions to reasonable alternative prior choices without prohibitive computational expense. The authors note that the learned harmonic mean estimator is the key enabler of this post-hoc efficiency and that the method is applicable to any scenario where posterior samples are available, potentially extending to combining evidences across datasets.