Detecting Model Misspecification in Bayesian Inverse Problems via Variational Gradient Descent

1. Problem Statement

Bayesian inference is optimal when the statistical model is well-specified (i.e., the true data-generating process is contained within the model family). However, in model misspecification scenarios, standard Bayesian inference can fail catastrophically: the posterior distribution may concentrate tightly around a "best-fit" parameter that is still incorrect, leading to over-confident and erroneous predictions.

Existing methods for detecting misspecification face significant limitations:

• Predictive approaches (e.g., posterior predictive checks, cross-validation) often require holding out data, which is difficult for dependent data (common in physics-based inverse problems) and computationally expensive.
• Comparative approaches (e.g., model selection) require manually constructing alternative models, which is infeasible when the physical mechanisms are complex or poorly understood.

The authors aim to develop a general, automated, and computationally efficient method to detect model misspecification without requiring data splitting or manual construction of alternative models.

2. Methodology

The proposed framework combines Predictively Oriented (PrO) inference with Variational Gradient Descent (VGD).

A. Predictively Oriented (PrO) Posterior

Instead of restricting inference to a parametric family $P_\theta$, the authors "lift" the model to an infinite mixture model:
$$P_Q(y|x) = \int P_\theta(y|x) dQ(\theta)$$
where $Q$ is a mixing distribution over the parameter space.

• Standard Bayesian Posterior ($Q_{Bayes}$): Minimizes the negative log-likelihood plus KL-divergence from the prior.
• PrO Posterior ($Q_{PrO}$): Minimizes an entropy-regularized mixture log-likelihood:
  $$Q_{PrO} := \arg\min_{Q} \left( -\sum_{i=1}^n \log \int p_\theta(y_i|x_i) dQ(\theta) + \text{KLD}(Q || Q_0) \right)$$
  • Intuition: If the model is well-specified, $Q_{PrO}$ will concentrate around the true parameter (similar to $Q_{Bayes}$). If the model is misspecified, $Q_{PrO}$ will learn a non-trivial mixture distribution to better explain the data, resulting in higher predictive uncertainty compared to the over-confident $Q_{Bayes}$.

B. Variational Gradient Descent (VGD)

Since $Q_{PrO}$ is defined as the minimizer of a non-linear variational objective, standard MCMC methods are not directly applicable. The authors propose using Variational Gradient Descent (VGD), a particle-based optimization algorithm.

• Mechanism: VGD evolves a set of particles $\{\theta_j\}$ in the parameter space to minimize the objective functional.
• Algorithmic Efficiency: The update rule for $Q_{PrO}$ is a one-line modification of the standard Stein Variational Gradient Descent (SVGD) used for $Q_{Bayes}$. Specifically, the gradient term includes a weighting factor $w_\theta = p_\theta(y|x) / p_Q(y|x)$, making the two posteriors computable using identical code structures.
• Theoretical Guarantee: The paper establishes novel consistency conditions for VGD applied to the PrO objective, proving that the particle approximation converges to the true minimizer under specific regularity conditions (e.g., bounded Fisher scores, which hold in physics-constrained problems).

C. Hypothesis Testing for Misspecification

To operationalize the detection of misspecification, the authors propose a formal test comparing the predictive distributions of $Q_{Bayes}$ and $Q_{PrO}$:

1. Test Statistic: Compute the Maximum Mean Discrepancy (MMD) between the predictive distributions $P_{Bayes}$ and $P_{PrO}$.
2. Null Distribution: Use a parametric bootstrap. Simulate synthetic datasets from the estimated "best" parameter $\hat{\theta}$ (under the assumption of a well-specified model) and re-compute the MMD statistic.
3. Decision: If the observed MMD between the real $P_{Bayes}$ and $P_{PrO}$ falls in the tail of the bootstrap null distribution, the null hypothesis (that the model is well-specified) is rejected.
4. Theoretical Validity: The paper proves the asymptotic correctness of this parametric bootstrap null under strong log-concavity and Lipschitz conditions.

3. Key Contributions

1. Unified Detection Framework: A method that acts as both a predictive check (assessing predictive performance) and a comparative approach (automatically generating a candidate set of models via the mixture lifting) without manual intervention.
2. Efficient Numerical Algorithm: The demonstration that $Q_{PrO}$ can be computed via VGD with only a minimal code change from standard SVGD, enabling parallel computation of both posteriors.
3. Theoretical Foundations:
   • Proof of the well-definedness of the PrO posterior.
   • Novel consistency results for VGD in the context of non-linear PrO objectives.
   • Proof of the asymptotic correctness of the parametric bootstrap test for misspecification.
4. Diagnostic Tool for Inverse Problems: A visual and statistical diagnostic that distinguishes between well-specified and misspecified scenarios in high-dimensional, physics-constrained inverse problems where traditional cross-validation is impractical.

4. Results

The authors validated their approach through two main studies:

A. Simulation Study (Gaussian Regression)

• Setup: Various regression tasks (linear, quadratic, sigmoid) with both well-specified and misspecified data generation.
• Findings:
  • In well-specified cases, $Q_{Bayes}$ and $Q_{PrO}$ were nearly identical, and the MMD statistic was low.
  • In misspecified cases, $Q_{PrO}$ adapted by spreading out (learning a mixture), while $Q_{Bayes}$ remained concentrated on a suboptimal parameter.
  • The MMD test successfully rejected the well-specified null hypothesis with high power as sample size increased.
  • Detection power decreased as parameter dimension increased but remained effective even at $d=50$.

B. Case Study: Seismic Travel Time Tomography

• Context: Reconstructing subsurface velocity fields from seismic wave arrival times. This involves solving the Eikonal equation (a PDE), making likelihood evaluation computationally expensive.
• Scenario: A realistic misspecification was introduced where the sensor locations in the statistical model were incorrect, though the data were generated with correct sensor locations.
• Findings:
  • Visual Diagnostic: In the well-specified case, the reconstructed velocity fields (mean and variance) from $Q_{Bayes}$ and $Q_{PrO}$ were indistinguishable.
  • Misspecification Detection: When sensor locations were misspecified, $Q_{Bayes}$ produced an over-confident reconstruction near the sensors, whereas $Q_{PrO}$ correctly identified the uncertainty and produced a different, more robust reconstruction.
  • Computational Cost: The diagnostic plot required only a factor of 2 increase in computational cost compared to running standard Bayesian inference alone.

5. Significance

This paper addresses a critical gap in Bayesian statistics: how to detect when a model is wrong without knowing how to fix it.

• Practicality: It offers a "drop-in" solution for complex inverse problems (like seismology) where constructing alternative physical models is difficult and cross-validation is computationally prohibitive.
• Robustness: By utilizing the PrO posterior, the method provides a safety net against over-confidence. If misspecification is detected, the PrO posterior itself can be used as a more reliable predictive model than the standard Bayesian posterior.
• Scalability: The use of VGD allows the method to scale to high-dimensional parameter spaces and complex likelihoods (PDE-constrained) where traditional sampling methods fail.

In summary, the authors provide a rigorous, computationally efficient, and automated framework for model criticism in Bayesian inference, leveraging the power of variational optimization to detect and mitigate the risks of model misspecification.