Technical Summary: Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantification

Problem Statement

Approximate Bayesian inference (ABI) traditionally focuses on computing the posterior distribution of model parameters, $p(\theta | D)$. However, in practical applications, the primary object of interest is often the posterior predictive distribution, $p(\hat{y}_{n+1} | x_{n+1}, D)$, which quantifies the uncertainty of predictions rather than the parameters themselves.

Standard methods for approximating the posterior predictive, such as Variational Inference (VI), Markov Chain Monte Carlo (MCMC), or classical Laplace approximations applied to the parameter posterior followed by Monte Carlo (MC) sampling, often incur significant computational overhead. Specifically, in Bayesian Neural Networks (BNNs), the necessity of MC sampling to integrate over the parameter posterior drastically increases inference costs. Furthermore, existing deterministic alternatives often rely on strong distributional assumptions or restrict uncertainty modeling to the final layer, limiting their applicability to general deep regression models.

Methodology

The authors propose Self-Supervised Laplace Approximation (SSLA), a technique that bypasses the explicit computation of the parameter posterior to directly approximate the posterior predictive distribution. The method is inspired by self-training and semi-supervised learning concepts, specifically the idea of learning from self-predicted data.

Core Mechanism

The fundamental insight of SSLA is to quantify predictive uncertainty by observing how a model's fit changes when it is "refit" on its own self-predicted data.

1. Augmented Dataset: Given a training dataset $D$ and a new input $x_{n+1}$, the model generates a point prediction $\hat{y}_{n+1}$ (e.g., the MAP prediction). An augmented dataset $D_{n+1} = D \cup \{(x_{n+1}, \hat{y}_{n+1})\}$ is constructed.
2. Refitting: The model is refit on $D_{n+1}$ to find a new optimum $\tilde{\theta}$.
3. Uncertainty Quantification: The predictive uncertainty is derived from the change in the model's likelihood, prior, and curvature (Fisher information) induced by the inclusion of the self-prediction.
   • If the model assigns high likelihood to its self-prediction (i.e., $\tilde{\theta} \approx \hat{\theta}$), the prediction is considered low uncertainty.
   • If the self-prediction causes a significant shift in the model fit (large change in likelihood or curvature), the prediction is assigned high uncertainty.

Mathematical Formulation

The authors derive a tractable approximation for the log-posterior predictive density, $\log p(\hat{y}_{n+1} | x_{n+1}, D)$, which decomposes into three interpretable increments evaluated at the model fits:
$$\log p(\hat{y}_{n+1} | x_{n+1}, D) \approx \underbrace{[\tilde{\ell}(\tilde{\theta}) - \ell(\hat{\theta})]}_{\text{Likelihood Increment}} + \underbrace{[\log \pi(\tilde{\theta}) - \log \pi(\hat{\theta})]}_{\text{Prior Increment}} - \underbrace{\frac{1}{2}[\log |\tilde{J}(\tilde{\theta})| - \log |J(\hat{\theta})|]}_{\text{Curvature Increment}}$$
Where:

• $\hat{\theta}$ is the MAP estimate on $D$.
• $\tilde{\theta}$ is the MAP estimate on $D \cup \{(x_{n+1}, \hat{y}_{n+1})\}$.
• $\ell$ and $\tilde{\ell}$ are log-likelihoods.
• $J$ and $\tilde{J}$ are observed Fisher information matrices (negative Hessians).

Approximate Variant (ASSLA)

To eliminate the computational cost of refitting the model (solving for $\tilde{\theta}$), the authors introduce Approximate SSLA (ASSLA).

• Asymptotic Expansion: Theoretical results (Lemma 1, Theorem 1) show that for large $n$, the difference between the augmented optimum $\tilde{\theta}$ and the original optimum $\hat{\theta}$ is of order $O(n^{-1})$.
• Simplification: ASSLA approximates $\tilde{\theta} \approx \hat{\theta}$ and retains only leading-order terms. This allows the method to evaluate the predictive uncertainty using only the original fit $\hat{\theta}$ and the local curvature contributions of the new pseudo-observation, avoiding the need for a second optimization step.
• Prior Modularity: In the ASSLA approximation, the prior increment is dropped as a higher-order term, making the method computationally efficient while retaining sensitivity to likelihood and curvature changes.

Key Contributions

1. Direct Predictive Approximation: A novel framework that approximates the posterior predictive distribution directly, bypassing the parameter posterior and the need for Monte Carlo sampling.
2. Self-Supervised Mechanism: The introduction of a self-training based Laplace approximation where uncertainty is quantified via the sensitivity of the model fit to self-predicted data.
3. Modularity: The SSLA formulation allows for the plug-in of different prior specifications without retraining, facilitating Bayesian sensitivity analysis regarding prior choice.
4. Computational Efficiency: The derivation of ASSLA, which removes the refitting requirement, making the method scalable to large models and datasets while maintaining competitive calibration.
5. Theoretical Justification: Rigorous proofs characterizing the approximation error of ASSLA, demonstrating that the error decreases linearly with sample size $n$ under standard regularity conditions.

Experimental Results

The authors evaluated (A)SSLA on regression tasks ranging from conjugate analytic models to complex heteroscedastic regression with neural networks.

• Conjugate Models: In Normal-Normal and Poisson-Gamma settings, SSLA and ASSLA closely matched the analytically computed posterior predictive distributions across sample sizes from $n=20$ to $n=1,000,000$. Numerical instabilities in ASSLA were observed only at very large $n$ ($>10^6$) due to floating-point cancellation, which was resolved using double precision.
• Heteroscedastic Regression (Synthetic): On synthetic data with input-dependent noise, ASSLA produced smoother and tighter credible intervals than classical Laplace approximations (which were overly conservative) and competitive with Variational Inference (VI) and Hamiltonian Monte Carlo (HMC). However, ASSLA showed a tendency to slightly underestimate uncertainty in wide intervals compared to HMC.
• Real-World Benchmarks: Evaluated on diverse UCI datasets (e.g., Concrete, Wine Quality, Bike Sharing):
  • Calibration: ASSLA generally achieved better Negative Log-Likelihood (NLL) and Continuous Ranked Probability Score (CRPS) than classical Laplace, indicating sharper and better-calibrated predictions.
  • Coverage: While ASSLA sometimes exhibited under-coverage in difficult regimes (e.g., Liver Disorders dataset with poor signal-to-noise), it often outperformed the conservative Laplace baseline which produced overly wide intervals.
  • Efficiency: ASSLA provided a favorable trade-off between computational cost and predictive quality, avoiding the high overhead of MC sampling.

Significance and Claims

The paper claims that (A)SSLA offers a deterministic, sampling-free alternative to standard Bayesian inference techniques for predictive uncertainty quantification. The significance lies in:

• Bypassing Intractability: It avoids the computational bottleneck of integrating over the parameter space via MC sampling, which is a major limitation in Bayesian Deep Learning.
• Practical Applicability: The method is applicable to generic deep regression models and supports modular prior specification, which is particularly relevant for emerging fields like Credal Bayesian Deep Learning.
• Trade-off Management: The authors position ASSLA as a method that balances computational tractability with competitive probabilistic calibration, offering a practical solution for real-world regression tasks where full Bayesian inference is too expensive.

The authors remain modest regarding limitations, acknowledging that SSLA can become unstable on very large datasets or with prior-data conflicts, and that ASSLA may underestimate uncertainty in high-variance regions. They suggest that these methods should be used alongside drift/OOD detection and robust likelihood modeling.