Generative Bayesian Inference with GANs

1. Problem Statement

The paper addresses the challenge of likelihood-free Bayesian inference. In many complex scientific applications (e.g., epidemiology, ecology), the data-generating process is defined by a simulator model that is easy to simulate but possesses an intractable likelihood function (i.e., $p(X|\theta)$ cannot be evaluated analytically).

• Current Limitations: Traditional Bayesian inference requires evaluating the likelihood. Approximate Bayesian Computation (ABC) is the standard alternative, which approximates the posterior by accepting parameters that generate "fake" data close to observed data. However, standard ABC (especially rejection ABC) is computationally inefficient, often requiring massive numbers of simulations to find plausible parameters, particularly when the posterior is much narrower than the prior.
• Goal: To develop a method that efficiently simulates from the posterior distribution $\pi(\theta | X_0)$ without evaluating the likelihood, leveraging the power of deep learning while maintaining theoretical guarantees.

2. Methodology

The authors propose a framework called Bayesian GAN (B-GAN), which bridges Approximate Bayesian Computation (ABC) with Generative Adversarial Networks (GANs) and Adversarial Variational Bayes.

Core Concept: Conditional GANs for Posterior Sampling

Instead of using GANs to learn a likelihood (generating data from parameters), the authors use Conditional GANs (cGANs) to learn a mapping from noise to parameters, conditioned on observed data.

• The Setup:
  • Generator ($g_\beta$): A neural network that takes a noise vector $Z$ and observed data $X$ as input, outputting a parameter sample $\theta = g_\beta(Z, X)$. The goal is to learn the conditional distribution $\pi(\theta | X)$.
  • Critic/Discriminator ($f_\omega$): A neural network that distinguishes between real parameter-data pairs $(\theta, X)$ drawn from the joint prior-simulator distribution and fake pairs $(g_\beta(Z, X), X)$ generated by the generator.
• Training Mechanism: The method uses Wasserstein GANs (WGAN) with a gradient penalty to ensure stability. The training relies on an ABC Reference Table consisting of $T$ simulated pairs $\{(\theta_j, X_j)\}$ drawn from the prior $\pi(\theta)$ and the simulator $P_\theta$. Crucially, the observed data $X_0$ is not used during the training phase; it is only used during the simulation phase to condition the generator.

Refinement Strategies

To improve accuracy for specific observed data $X_0$, the authors propose two post-processing/local refinement strategies:

1. Two-Step Refinement (B-GAN-2S / Reinforcement Learning):

   • Step 1: Train a "pilot" B-GAN on the original prior to get a rough approximation of the posterior.
   • Step 2: Use the pilot generator to create a new, more informative proposal distribution (an empirical Bayes prior) that is concentrated near $X_0$.
   • Correction: Simulate a new reference table using this proposal and retrain the generator. The resulting samples are importance re-weighted to correct for the change in prior, yielding samples from the true posterior.

2. Adversarial Variational Bayes (B-GAN-VB):

   • This approach integrates the observed data $X_0$ directly into the training objective.
   • It modifies the generator's loss function to include a term that maximizes the Evidence Lower Bound (ELBO) for $X_0$.
   • Since the likelihood is intractable, the ELBO is approximated using a contrastive learning approach (a learned critic) to estimate the density ratio, effectively turning the problem into a variational optimization regularized by the adversarial loss.

Handling High-Dimensional Data

The paper addresses the "curse of dimensionality" for large datasets (large $n$) by proposing:

• Sequential Updates: Processing data in batches, using the posterior of one batch as the prior for the next.
• Deep Set Architecture: Utilizing permutation-invariant neural networks (Deep Sets) to handle exchangeable i.i.d. data efficiently, avoiding the need to flatten large data vectors into massive input dimensions.

3. Key Contributions

1. Novel Framework (B-GAN): The first method to explicitly frame likelihood-free Bayesian inference as a conditional GAN problem, learning a deterministic mapping from noise to the posterior.
2. Theoretical Guarantees: The authors provide frequentist-Bayesian theoretical results. They prove that the expected squared Total Variation (TV) distance between the true posterior and the B-GAN approximation converges to zero as the sample size $n \to \infty$, provided the neural networks are sufficiently expressive and the ABC reference table is large enough.
3. Efficiency Improvements: The introduction of the Two-Step and Variational refinements significantly reduces the approximation gap compared to vanilla B-GAN, allowing for high-quality posterior reconstruction with fewer samples than standard ABC.
4. Scalability: Demonstrated ability to handle complex, high-dimensional, and non-i.i.d. data structures (e.g., time series) through Deep Set architectures and sequential updates, overcoming limitations of previous likelihood-free methods.

4. Results and Performance

The methods were evaluated on several simulated examples and a real-world application:

• Toy Gaussian Example: B-GAN successfully captured multimodal posteriors where standard ABC failed. The Two-Step refinement (B-GAN-2S) achieved the lowest Maximum Mean Discrepancy (MMD) compared to the truth.
• Lotka-Volterra Model (Ecology): A classic ODE-based model with intractable likelihood. B-GAN-2S and B-GAN-VB outperformed Sequential Neural Likelihood (SNL) and standard ABC in terms of bias, credible interval width, and posterior mass concentration.
• Boom-and-Bust Model: A challenging discrete stochastic process. B-GAN variants provided tighter credible intervals and better parameter recovery than competitors, which struggled with the non-normality of the statistics.
• SIR Epidemic Model (Real Data): Applied to the 1967 Tristan da Cunha common cold outbreak. B-GAN-VB provided the tightest credible intervals and the best fit to the observed infection/recovery data compared to other methods.
• Computational Cost: While training requires significant GPU resources, the sampling cost is negligible once the generator is trained. The methods scale better than Wasserstein ABC (W2) and standard ABC (SS) for high-dimensional data.

5. Significance

This paper represents a significant advancement in simulation-based inference (SBI).

• Bridging Communities: It successfully unifies the fields of Approximate Bayesian Computation and Generative Adversarial Networks, showing that GANs can be repurposed for statistical inference rather than just image generation.
• Practical Utility: It offers a practical solution for practitioners dealing with complex simulator models where likelihoods are unavailable, providing a tool that is both theoretically grounded and empirically competitive with state-of-the-art methods like SNL.
• Theoretical Rigor: By establishing convergence rates and error bounds for neural network-based posterior samplers, the paper moves the field beyond heuristic applications of deep learning in statistics, providing a solid foundation for future research in implicit Bayesian inference.

In summary, Wang and Ročková present a robust, theoretically supported, and highly competitive framework for performing Bayesian inference in the absence of tractable likelihoods, leveraging the expressive power of deep generative models to overcome the inefficiencies of traditional simulation-based methods.