A Convex Approximation Framework for Neural Likelihood-Based Bayesian Inverse Problems

Technical Summary: A Convex Approximation Framework for Neural Likelihood-Based Bayesian Inverse Problems

Problem Statement
Many scientific and engineering inverse problems are hindered by the difficulty of accurately modeling physical mechanisms, quantifying measurement uncertainty, or the prohibitive computational cost of high-fidelity simulations (e.g., solving partial differential equations). These challenges render classical probabilistic inference methods, such as Markov Chain Monte Carlo (MCMC), infeasible, particularly in high-dimensional or infinite-dimensional settings (e.g., Bayesian inverse problems on function spaces). While machine learning offers an alternative to explicit parametric modeling, existing approaches to neural likelihood approximation—learning the likelihood function directly from data without knowledge of the underlying data-generating process—often rely on restrictive model classes. Specifically, standard methods that minimize the Kullback–Leibler (KL) divergence between the true and approximate posterior often require the likelihood surrogate to be explicitly normalized (e.g., via normalizing flows or invertible networks). These constraints can limit the expressiveness of the model, potentially excluding near-optimal likelihood functions.

Methodology
The authors propose a framework that reformulates neural likelihood approximation to work with un-normalized potentials (negative log-likelihoods, or NLLs), folding the normalization requirement directly into the training objective.

1. Objective Function Reformulation:
   Instead of minimizing the KL divergence between posteriors directly, the authors derive an equivalent objective function $\Phi(f)$ based on the expected KL divergence between the true posterior $\mu_y$ and an approximate posterior $\mu_y^f$ induced by an un-normalized potential $f$:
   $$\Phi(f) = \mathbb{E}_{\pi}[\text{KL}(\mu_y \| \mu_y^f)] = C + \mathbb{E}_{\lambda}[f(x; y)] + \mathbb{E}_{\pi}[\log Z_f(y)]$$
   where $Z_f(y) = \int_H \exp(-f(x; y)) \mu(dx)$ is the normalization constant, $\lambda$ is the joint distribution of data $(x, y)$, and $C$ is a constant independent of $f$. This formulation allows the objective to be evaluated using samples from the joint distribution without requiring access to the true potential $L$ or explicit normalization of the network output.

2. Convexity Analysis:
   A central theoretical contribution is the proof that the objective function $\Phi(f)$ is strictly convex with respect to the un-normalized potential $f$ (Theorem 2.8). This holds provided the function class $\mathcal{F}$ is convex. This contrasts with standard normalizing flow parametrizations, where the set of functions integrating to one is generally non-convex. The strict convexity ensures the existence of a unique minimizer in the equivalence class of potentials (where functions differing only by a $y$-dependent additive constant are equivalent).

3. Data-Driven Estimation and Consistency:
   The authors propose a data-driven estimator $\Phi_N$ using a double-loop Monte Carlo approach to approximate the nested expectation involving the normalization constant $Z_f(y)$. They establish the consistency of this estimator (Theorem 3.7), proving that as the sample size $N \to \infty$, the empirical minimizer converges in probability to the true potential $L$, under assumptions regarding the prior distribution and the regularity of the function class (e.g., bounded subsets of Hölder spaces).

Key Contributions

• Convex Optimization Framework: The paper demonstrates that by optimizing over un-normalized potentials and absorbing normalization into the loss, the learning problem becomes strictly convex. This removes the need for restrictive architectural constraints (like normalizing flows) to enforce probability measure properties.
• Theoretical Guarantees: The authors provide rigorous proofs for the convexity of the objective and the consistency of the empirical minimizer, establishing that the learned potential converges to the true potential as data increases.
• Data Efficiency: The framework requires only joint data $(x, y)$ from the simulator (or physical system) and does not require knowledge of the forward map, the noise level, or evaluations of the true likelihood function.

Numerical Results
The framework was validated on two Bayesian inverse problems:

1. Deblurring Problem: A toy problem involving a 1D Gaussian process. The neural likelihood approximations (free-form, residual, and calibrated residual) produced posterior means and confidence intervals very close to the ground truth. Notably, the free-form approximation (which does not assume known noise levels) outperformed Gaussian Process (GP) regression, even when the GP had access to evaluations of the true likelihood.
2. Semiconductor Doping Profile: A non-linear, PDE-based imaging problem estimating the doping profile from voltage-current measurements.
   • Computational Speed: The NLL approximations were approximately 200 times faster than the finite element PDE solver for single evaluations on a CPU, and up to 20,000 times faster under parallelization on a GPU.
   • Accuracy: The free-form and residual approximations yielded posterior statistics (bias and variance) nearly identical to the exact PDE-based posterior. The calibrated residual approximation (which assumes known noise) performed well, whereas the standard residual approximation struggled when noise levels were unknown.
   • Efficiency: While training and data generation took roughly 6 hours, this allowed for the generation of posterior samples that would have taken 265 hours using the direct PDE model.

Significance and Claims
The paper claims to provide a robust theoretical foundation for neural likelihood approximation that overcomes the limitations of restrictive parametric classes. By establishing convexity and consistency, the work enables offline learning purely based on empirical data, which can subsequently accelerate posterior sampling in scenarios where the forward model is unknown or computationally expensive.

The authors note specific limitations:

• The consistency results are currently established in a finite-dimensional setting due to the reliance on existing empirical process theory (bracketing entropy) developed for Euclidean spaces. Extending these guarantees to fully infinite-dimensional function classes remains an open direction.
• Quantitative convergence rates for the estimators are not provided, as the nested Monte Carlo structure complicates the application of classical M-estimator rate-of-convergence results.
• While the optimization in the function space is convex, the optimization in the neural network parameters remains non-convex; however, the strict convexity of the population objective ensures a unique target for the learning process.