Calibrating Bayesian Inference

1. Problem Statement

The paper addresses a critical vulnerability in "pragmatic" Bayesian inference, which is widely used in fields like psychology. While Bayesian methods offer intuitive uncertainty quantification, they often rely on prior distributions that are chosen for computational convenience (e.g., default or conjugate priors) rather than representing genuine subjective beliefs.

• The Core Issue: Standard Bayesian validity relies on the assumption that the prior used for inference matches the true data-generating process (the "true prior"). When this assumption is violated (prior misspecification), Bayesian inference can produce false confidence.
• False Confidence Theorem (FCT): The authors highlight that if the true parameter is fixed (a point mass) but the analyst uses a non-degenerate prior, the resulting posterior can assign high probability to false hypotheses.
• Limitations of Current Solutions:
  • Large-sample theory: Asymptotic results (e.g., Bernstein-von Mises theorem) often fail in small-sample scenarios common in psychology.
  • Sensitivity analysis: Testing a few priors is often inconclusive and computationally expensive.
  • Monte Carlo studies: Existing simulation studies are often model-specific and not generalizable.

The authors argue that to ensure reliability, Bayesian procedures used pragmatically must satisfy frequentist validity: the guarantee that the inference procedure does not systematically produce misleading results over repeated sampling, regardless of the true parameter-generating mechanism.

2. Methodology

The paper proposes a computational framework to calibrate posterior-based inference to achieve frequentist validity.

A. Theoretical Framework

• Posterior Possibility Contours: Instead of relying solely on posterior probabilities, the authors utilize a posterior possibility contour ($\varpi_y(\theta)$). This function summarizes nested credible regions and induces a possibility measure that upper-bounds posterior probabilities.
• Frequentist Validity Criterion: A procedure is valid if, for any fixed true parameter $\theta$, the probability that the possibility measure assigns a value $\le \alpha$ is at most $\alpha$.
  $$\sup_{\theta \in \Theta} P_{Y|\theta}\{ \pi_Y(\theta) \le \alpha \} \le \alpha$$
  where $\pi_y(\theta)$ is the calibrated possibility contour.
• The Calibration Problem: The goal is to transform the standard posterior contour into a calibrated contour $\tilde{\varpi}_y(\theta)$ by adjusting the threshold of the test statistic. This involves solving an optimization problem to find the maximum p-value on the boundary of a credible region:
  $$\alpha^*_y(\xi) = \sup_{\theta \in \partial D_\xi(y)} \pi_y(\theta)$$
  where $\partial D_\xi(y)$ is the boundary of the credible region defined by test statistic threshold $\xi$.

B. The Algorithm: SPRSA

To solve the constrained optimization problem (finding the maximum p-value on a manifold boundary), the authors propose a novel Simultaneous-Perturbation Riemannian Stochastic Approximation (SPRSA) algorithm.

• Manifold Optimization: The boundary of the credible region is treated as a differentiable submanifold. The algorithm uses Riemannian gradient ascent, projecting the gradient of the p-value function onto the tangent space of this manifold.
• Gradient-Free Approximation: Since the p-value function involves an integral that is difficult to differentiate analytically, the algorithm uses Simultaneous Perturbation (SP) to estimate the gradient. This involves perturbing all parameters simultaneously with random noise, requiring only two evaluations of the test statistic per iteration regardless of dimension.
• Retraction: A retraction map is used to ensure that updated parameter estimates remain on the manifold constraint.
• Averaging: To improve convergence, the algorithm employs recursive averaging of iterates (Polyak-Ruppert style) adapted for Riemannian manifolds.

3. Key Contributions

1. Theoretical Justification: The paper rigorously demonstrates that frequentist validity is a stronger condition than Bayesian validity and implies it for any parameter-generating prior. It establishes that uncalibrated Bayesian inference is vulnerable to false confidence when priors are misspecified.
2. Novel Algorithm: The development of the SPRSA algorithm is a significant methodological contribution. It combines manifold optimization with gradient-free stochastic approximation, making it scalable to realistic model sizes where calculating exact gradients is intractable.
3. Practical Calibration Procedure: The authors provide a concrete workflow for calibrating credible regions (elliptical, HPD, and symmetric intervals) based on Wald, Posterior Density Ratio (PDR), and marginal Wald statistics.

4. Results

The authors conducted a Monte Carlo experiment using Gaussian location-scale regression (a model where both mean and variance depend on covariates) under various conditions:

• Scenarios:
  • Scenario 1: Non-degenerate parameter-generating prior (mismatched with inference prior).
  • Scenario 2: Fixed true parameters (point-mass prior, representing the frequentist setup).
  • Scenario 3: Correctly specified strong prior.
• Findings:
  • Uncalibrated Inference: Standard Bayesian inference (using MCMC or asymptotic chi-square approximations) was frequently liberal (invalid), particularly in small samples ($n=100$) with many covariates ($m=10$) or under fixed true parameters. The chi-square approximation was consistently too liberal.
  • MCMC Issues: MCMC sampling showed high non-convergence rates in complex scenarios (e.g., heavy-tailed data with weak priors).
  • Calibrated Inference: The proposed SPRSA calibration consistently maintained frequentist validity across all scenarios, including those where standard methods failed. The calibrated contours were generally wider (more conservative) than uncalibrated ones, correctly reflecting the uncertainty.
• Empirical Example: Applied to the Ginzberg depression dataset, the calibration procedure produced asymmetric and wider credible intervals compared to standard methods, highlighting the potential for misleading precision in uncalibrated Bayesian analysis.

5. Significance

• Safety in Pragmatic Bayes: The paper provides a "safety net" for researchers who use Bayesian methods instrumentally without strong prior beliefs. It offers a way to ensure that their uncertainty quantification is statistically valid in the long run.
• Bridging Paradigms: It bridges the gap between Bayesian and frequentist paradigms by showing that Bayesian procedures can be tuned to satisfy frequentist validity criteria, thereby enhancing the replicability and reliability of statistical conclusions in psychology and other social sciences.
• Computational Feasibility: By demonstrating that complex calibration is achievable via SPRSA, the paper moves the concept of "calibrated Bayes" from theoretical ideal to practical application for realistic, high-dimensional models.

Conclusion: The authors conclude that while Bayesian inference is powerful, its reliance on priors makes it susceptible to false confidence. They recommend calibrating posterior-based inference using their proposed algorithm to guarantee frequentist validity, ensuring that statistical conclusions remain robust regardless of the true data-generating mechanism.