Technical Summary: Onsager–Machlup Posterior Transport for Deep Gaussian Processes

1. Problem Statement

Deep Gaussian Processes (DGPs) offer a Bayesian deep learning framework with calibrated uncertainty by stacking Gaussian process (GP) layers. However, inference remains a central computational bottleneck. The posterior distribution over per-layer inducing variables $U = \{U^{(\ell)}\}_{\ell=1}^L$ is non-Gaussian and highly correlated across layers, rendering exact inference intractable.

Existing approximate inference methods face a trade-off:

• Explicit Density Fitting (ELBO-based): Methods like Doubly-Stochastic Variational Inference (DSVI) enforce a Gaussian factorization, which is restrictive for multimodal or correlated posteriors.
• Score-Based SDEs: Methods like DDVI and DBVI define the posterior as the terminal of a reverse-time Stochastic Differential Equation (SDE) trained via denoising score matching (DSM) and a Girsanov-form Evidence Lower Bound (ELBO). While flexible, these require training auxiliary score networks and suffer from the variance of stochastic integration.
• Sampling: Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) is computationally expensive and struggles to scale to large datasets.

The core challenge is to perform inference on DGP inducing variables without the restrictive Gaussian assumptions of DSVI, while avoiding the high-variance auxiliary losses and stochastic integration costs of score-based SDE methods.

2. Methodology: FBVI-bridge-Path (OM-Path)

The authors propose posterior transport as an alternative formulation. Instead of fitting a density $q_\phi(U)$ via ELBO maximization, they learn a deterministic sampler $T_\phi$ that maps a tractable reference measure to posterior-relevant inducing variables.

2.1 From SDE to Deterministic ODE

The method builds upon the Doob-bridged forward SDE used in DBVI, which anchors the diffusion process to a data-dependent initial distribution via an amortizer $\mu_\theta(x)$.

• Reference Process: The authors apply Song et al.'s (2021) probability-flow ODE reduction to the DBVI bridge SDE. This yields a closed-form reference drift $v_{ref}^{Bri}(U, s)$ derived directly from the bridge marginal coefficients (mean and variance), eliminating the need for score matching.
• Deterministic Transport: The variational object is a deterministic ODE $\dot{U}_\tau = v_\phi(U_\tau, 1-\tau, \text{ctx})$ that transports samples from the noise-side marginal of the bridge to the posterior.

2.2 The Onsager–Machlup Objective

A critical theoretical insight is that the standard Girsanov KL divergence used in SDE-based VI diverges as the diffusion coefficient $g \to 0$ (the deterministic limit). The correct objective for the deterministic limit is the Onsager–Machlup (OM) action, which corresponds to the Freidlin–Wentzell large deviation rate functional.

The training objective $L_{OM}$ is defined as:
$$L_{OM}(\phi) = -\mathbb{E}_{q_\phi}[\log p(y | F(L))] + \frac{\alpha}{2} \int_0^1 \mathbb{E} \left[ \| v_\phi(U_\tau, 1-\tau, \text{ctx}) - v_{ref}^{Bri}(U_\tau, 1-\tau) \|^2 \right] d\tau$$

• Data Term: Negative log-likelihood of the endpoint.
• Path Prior: A trace-free regularizer measuring the deviation of the learned velocity field from the closed-form reference drift.
• Key Properties:
  • Trace-free: No Hutchinson trace estimation (unlike FFJORD/CNF).
  • Auxiliary-loss-free: No denoising score matching (unlike DDVI/DBVI).
  • Partition-function-free: No estimation of $\partial_t \log Z_t$ (unlike NFS2).

2.3 Theoretical Status

Theorem 1 establishes that $L_{OM}$ identifies with the small-noise Maximum A Posteriori (MAP) path of a tempered Doob-bridge path posterior via the Freidlin–Wentzell Large Deviation Principle (LDP) as $\epsilon \to 0$.

• At the finite training value ($\alpha=1, \epsilon=1$), the objective acts as a path-space regularizer for a Gaussian reference path measure tilted by the endpoint likelihood.
• The method is a path-space MAP estimator, not an ELBO. The authors explicitly note this distinction, contrasting it with strict lower-bound methods.

2.4 Strict-ELBO Ablations

The paper derives two strict path-space ELBO variants on the same bridge backbone:

1. FBVI-bridge-CNF: Uses the FFJORD instantaneous change-of-variables formula (Hutchinson trace) for the KL term.
2. FBVI-bridge-CNFOM: Adds the OM regularizer to the CNF objective.
   Empirically, these strict-ELBO variants underperform the trace-free MAP estimator due to the high variance introduced by the Hutchinson trace estimator.

3. Key Contributions

1. Posterior Transport for DGPs: Introduces a deterministic ODE-based posterior transport framework for DGP inducing variables, replacing ELBO-based score-SDE inference. The reference drift is closed-form, removing the need for score matching.
2. Finite-$\epsilon$ Path-Posterior Identity: Provides a rigorous identification of the training objective as the negative log unnormalized density of a tempered path posterior (finite $\alpha$) and connects it to the small-noise MAP path via Freidlin–Wentzell LDP (Theorem 1).
3. Bias-Variance Trade-off Analysis: Demonstrates that in the DGP posterior transport regime, reducing the variance of the path objective (by removing the Hutchinson trace) dominates the benefit of tracking an exact density. The trace-free MAP estimator outperforms strict ELBOs.
4. Empirical Validation: Shows statistically significant improvements over the state-of-the-art DBVI on large-scale UCI regression datasets ($N > 9,000$) and competitive performance on binary classification, while maintaining stability on small datasets.

4. Experimental Results

The method (FBVI-bridge-Path or OM-Path) was evaluated against DSVI, SGHMC, IPVI, DDVI, and DBVI on seven UCI regression datasets, large-scale regression (Airline, Year), and binary classification (SUSY, HIGGS).

• Small/Medium Regression (UCI):
  • Significant Wins: OM-Path achieves statistically significant wins (paired Wilcoxon test) over DBVI on the two largest datasets: Power ($p=0.014$) and Protein ($p=0.002$).
  • Ties: Statistical ties on Yacht and QSAR.
  • Losses: OM-Path concedes to DBVI on small-$N$ noisy datasets (Boston, Energy, Concrete), where the SDE-noise regularisation of DBVI's score parameterization provides a benefit.
• Large-Scale Regression: On the Year (515k rows) and Airline datasets, OM-Path achieves mean-best performance in RMSE and NLL, though statistical significance is not claimed due to limited seed counts.
• Binary Classification: OM-Path achieves tied-best performance with DBVI on SUSY and HIGGS, closing the gap previously seen with implicit-q surrogates.
• Few-Step Inference: Due to the deterministic nature of the ODE, OM-Path supports aggressive few-step inference (1–4 steps) with minimal accuracy degradation on datasets like Energy and Concrete, outperforming stochastic baselines in this regime.
• Bayesian Optimization: OM-Path provides superior uncertainty quantification on low-to-moderate dimensional synthetic benchmarks (Hartmann-6, Levy-20), outperforming DSVI.

5. Significance and Claims

The paper claims that FBVI-bridge-Path represents a rigorous, trace-free alternative to score-based variational inference for DGPs. Its significance lies in:

• Theoretical Rigor: Grounding the training objective in the Freidlin–Wentzell LDP and Onsager–Machlup action, providing a mathematically sound justification for deterministic posterior transport.
• Practical Efficiency: Eliminating the computational overhead of score matching and Hutchinson trace estimation, leading to faster training and more stable gradients.
• Performance: Demonstrating that a path-space MAP estimator, which trades the strict ELBO bound for lower variance, yields superior empirical results on large datasets where gradient variance is the limiting factor.

The authors are modest about their claims, acknowledging that:

• The method is not an ELBO and does not provide a lower bound on the marginal likelihood.
• Strict-ELBO variants exist but are empirically inferior in this specific regime due to variance.
• Performance on small, noisy datasets is currently best served by the stochastic regularisation of score-based methods (DBVI).
• The theoretical identification with the MAP path is an asymptotic result ($\epsilon \to 0$), while practical training occurs at finite $\epsilon$.