📊 statistics

Simulation-Based Empirical Bayes

This paper introduces Simulation-Based Empirical Bayes (SBEB), a novel framework that enables simultaneous inference for latent variables with implicit likelihoods by iteratively refining an amortized inference network to estimate the population prior without requiring an explicit density function.

Original authors: Xinwei Shen, Diana Cai, Cheng Zhang, David M. Blei

Published 2026-07-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinwei Shen, Diana Cai, Cheng Zhang, David M. Blei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but you don't have a clear rulebook for how the crime happened. Instead, you have a magical "what-if" machine. You can feed it a guess about the culprit's motive, and it spits out a fake crime scene. But here's the catch: the machine won't tell you how likely that motive was; it only shows you the result. This is the world of simulation-based inference. Scientists use these machines to study everything from how diseases spread to how people choose products, but because the math behind the scenes is too messy to calculate directly, they have to rely on running thousands of these "what-if" scenarios to figure out the truth.

Now, imagine you aren't just solving one case, but hundreds of similar ones at once—like figuring out the motives for 40 different cities during a measles outbreak. This is where Empirical Bayes comes in. Think of it as a smart detective who realizes that while every city is different, they all share a common "vibe" or pattern. Instead of guessing the rules for each city from scratch, the detective looks at all the cities together to learn what the "average" city looks like, then uses that knowledge to make better guesses for each specific one. It's like learning the general rules of soccer by watching a whole league, so you can better predict how a single player will move.

The big question this paper tackles is: What happens when you combine these two ideas? What if you have a magical "what-if" machine that you can't read the math of, and you need to solve hundreds of cases at once? Can you teach the detective to learn the "vibe" of the group while using the machine to solve the cases?

The authors of this paper, Xinwei Shen, Diana Cai, Cheng Zhang, and David M. Blei, say yes. They developed a new method called Simulation-Based Empirical Bayes (SBEB). It's a clever loop that lets the computer learn the "population rules" (the prior) directly from the data, even when the only way to understand the data is through a black-box simulator.

Here is how their magic trick works. Imagine you are trying to guess the secret settings of a video game character (like their speed or strength) based on how they move.

The Old Way (Fixed Prior): You start with a guess that "everyone is average." You run the simulator thousands of times with average settings to train a neural network (a smart AI) to guess the settings. But if the real players are actually all super-fast, your "average" guess is wrong, and your AI will be bad at predicting the fast players.
The New Way (SBEB): You start with that same "average" guess and train your AI. But then, instead of stopping, you ask the AI: "Hey, based on the real data we have, what do you think the settings actually are for these players?" The AI gives you a list of guesses. You then take those guesses, treat them as the new "average" settings, and feed them back into the simulator to generate new training data.
The Loop: You repeat this over and over. The AI gets better at guessing the settings, which helps you figure out the true "population average," which in turn helps the simulator generate better training data, which makes the AI even smarter. It's a self-improving cycle.

The paper proves that this loop works. In their theoretical analysis, they show that if the simulator is good and the math is perfect, this process will eventually find the true "population prior"—the exact set of rules that generated the data. They call this the "oracle" version, meaning it's the ideal scenario.

To see if it works in the real world, they tested it on five different simulated worlds, including a predator-prey ecosystem (like wolves and rabbits) and an evolutionary model. In every single case, their new SBEB method made more accurate guesses than the old "fixed prior" method. For example, in a linear Gaussian model, the old method had an error score of 0.410, while SBEB dropped it to 0.034—a massive improvement. In a more complex "predator-prey" simulation, the error dropped from 0.569 to 0.371.

They also tested it on two real-world datasets. First, they looked at how people choose digital cameras. When people only had a few choices to look at, the old method was okay, but as they saw more choices, SBEB learned the group's preferences and became the clear winner, predicting future choices much better. Second, they looked at measles outbreaks in 40 UK cities from 1944 to 1965. This is a classic case where the math is too hard to solve directly, so scientists use simulators. SBEB learned the common patterns of how measles spreads across cities and used that to predict future outbreaks more accurately than methods that assumed a fixed, unchanging starting point.

The paper doesn't claim this is a magic bullet that solves everything instantly. The authors are careful to note that SBEB is more computationally expensive—it takes more time and computing power because it has to run in rounds, constantly retraining and re-simulating. It also relies on the simulator being accurate; if the "what-if" machine is broken, the detective will learn the wrong rules. But for scientists dealing with complex, messy data where the math is hidden inside a simulator, this method offers a powerful new way to learn from the crowd while solving individual mysteries. It suggests that by letting the data teach us what the "average" looks like, we can get much closer to the truth than by sticking to our initial guesses.

Technical Summary: Simulation-Based Empirical Bayes

1. Problem Statement

The paper addresses the challenge of performing simultaneous inference across many related latent variables when the likelihood function $p(x | z)$ is intractable and available only through a simulator.

This problem combines two distinct statistical regimes:

Simulation-Based Inference (SBI): Used when the likelihood cannot be evaluated analytically. Standard SBI approximates the posterior $p(z | x)$ using an amortized inference network trained on simulated pairs $(z, x)$ . However, standard SBI requires a fixed, pre-specified prior $p(z)$ , which may not reflect the true population distribution of the latent variables.
Empirical Bayes (EB): Used to learn the prior $p(z)$ from a collection of related observations to improve simultaneous inference. Classical EB methods (e.g., for Gaussian means or Poisson counts) rely on tractable likelihoods to compute marginal likelihoods or posterior expectations.

The gap identified is that existing methods cannot simultaneously handle implicit likelihoods (requiring simulation) and learn the prior from the data. The authors aim to develop a framework that performs Empirical Bayes inference for models defined by complex simulators.

2. Methodology: Simulation-Based Empirical Bayes (SBEB)

The authors propose Simulation-Based Empirical Bayes (SBEB), an iterative algorithm that connects nonparametric EB with SBI. The core idea is to treat the population prior as a fixed point of an iterative process involving posterior averaging.

Theoretical Foundation

The method relies on the population prior identity:
$p(z) = \mathbb{E}_{p^*(x)} [p(z | x)]$
where $p^*(x)$ is the true population distribution of observations. This identity suggests that if one can approximate the posterior $p(z | x)$ for observed units, the population prior can be approximated by averaging these posteriors.

The Algorithm

SBEB operates in rounds ( $t = 1, \dots, T$ ) to refine both the inference network and the prior estimate:

Initialization: Start with an initial prior $p_0(z)$ and fit an amortized inference network $q(z; x, \phi_0)$ using standard fixed-prior SBI (Algorithm 1).
Iterative Refinement:
- Posterior Averaging: For the current iteration $t$ , compute an approximate population prior $\hat{p}_t(z)$ by averaging the approximate posteriors of the observed data units:
  $\hat{p}_t(z) = \frac{1}{n} \sum_{i=1}^n q(z; x_i, \hat{\phi}_{t-1})$
- Sampling from the Approximate Prior: Since $\hat{p}_t(z)$ $\overset{p}{^}_{t} (z)$ has no closed-form density, sampling is performed via a two-step process:
  1. Sample an observed unit $x_i$ uniformly from the dataset.
  2. Sample a latent parameter $z \sim q(z; x_i, \hat{\phi}_{t-1})$ .
- Simulation: Use the sampled $z$ to generate a new synthetic observation $x_{new} \sim p(x | z)$ using the simulator.
- Network Refitting: Train the inference network $\hat{\phi}_t$ on the new set of simulated pairs $(z, x_{new})$ to minimize the negative log-likelihood (standard SBI objective).
Output: After $T$ rounds, the algorithm returns the posterior approximations $\{q(z; x_i, \hat{\phi}_T)\}_{i=1}^n$ under the learned population prior.

Theoretical Guarantees

The paper analyzes an "oracle" version of SBEB where the inference network is exact and expectations are over the true population.

Monotonicity: Each iteration improves the fit of the induced marginal distribution $p_t(x)$ to the true population distribution $p^*(x)$ .
Convergence: Under a prior identifiability condition (where distinct priors induce distinct marginal distributions), the sequence of priors converges to the true population prior. Specifically, if the simulator is $\lambda$ -strongly prior-identifiable, the KL divergence between the estimated prior and the true prior decreases geometrically.

3. Key Contributions

Methodological Synthesis: The paper introduces SBEB, the first method to combine Empirical Bayes (learning the prior) with Simulation-Based Inference (handling intractable likelihoods).
Algorithmic Innovation: It proposes a practical iterative scheme that learns a flexible, nonparametric prior directly from data without requiring an explicit likelihood function, using an amortized inference network to facilitate sampling from the learned prior.
Theoretical Analysis: The authors provide convergence proofs for the oracle version of the algorithm, establishing that the method converges to the population prior under identifiability conditions.

4. Empirical Results

The authors evaluate SBEB across three settings:

A. Controlled Simulation Studies

Five simulators were tested: Linear Gaussian, Nonlinear static, Oscillator, Predator-prey, and Evolutionary (Wright-Fisher).

Metric: Mean Squared Error (MSE) of the posterior mean estimates against known ground-truth latent variables.
Findings: SBEB consistently outperformed fixed-prior SBI. In linear and oscillator models, SBEB nearly matched the performance of a "True Prior Oracle" (which uses the ground-truth prior). In complex nonlinear models, SBEB significantly reduced the error of fixed-prior SBI, though the oracle remained superior.

B. Heterogeneous Product Choice (Real Data)

Dataset: Discrete choice data from a digital camera survey (Allenby et al., 2014a,b).
Task: Infer person-specific preference vectors from partial choice data and predict held-out choices.
Findings: SBEB outperformed both fixed-prior SBI and individual Maximum A Posteriori (MAP) fits when the number of observed choices was moderate to large ( $R=9$ to $12$). The learned population prior effectively shrunk noisy individual estimates toward the population, improving predictive accuracy.

C. Measles Outbreaks in the U.K. (Real Data)

Dataset: Biweekly measles reports from 40 UK cities (1944–1965) analyzed via a stochastic epidemic simulator (TSIR-style).
Task: Infer city-specific epidemic parameters from a training window and reconstruct future disease burden.
Findings: SBEB achieved the lowest average error in predicting held-out log-incidence, particularly for cities with longer training histories. The learned EB prior allowed for more accurate simulation of future epidemics compared to fixed-prior SBI and per-city calibration.

5. Significance and Claims

The paper claims that SBEB extends the scope of Empirical Bayes from models with tractable likelihoods to those defined by scientific simulators.

Advancement over SBI: Unlike standard SBI, which relies on a fixed prior that may be misspecified, SBEB learns a data-driven prior that captures the population structure, leading to improved simultaneous inference accuracy.
Advancement over Classical EB: Unlike classical EB, which requires explicit likelihoods, SBEB operates in the "likelihood-free" regime, making it applicable to complex mechanistic models in epidemiology, ecology, and economics.
Trade-offs: The authors acknowledge that SBEB incurs higher computational costs than fixed-prior SBI (due to iterative rounds) and requires careful modeling of both the simulator and the amortized inference family. The method assumes the simulator provides an accurate conditional distribution; the practitioner remains responsible for the fidelity of the simulator and the expressiveness of the inference network.

In summary, SBEB provides a unified framework for simultaneous inference in complex, simulation-based scientific models, demonstrating that learning the prior from data can significantly enhance inference accuracy even when the likelihood is intractable.