An Old Look at Empirical Bayes

Technical Summary: An Old Look at Empirical Bayes

Problem Statement
The paper revisits Dennis Lindley's critique that empirical Bayes (EB) is statistically incoherent because it "uses the same data twice": first to estimate hyperparameters (or a prior) from the marginal distribution, and second to compute a posterior as if that prior were fixed and known. The authors argue that modern extensions of EB, specifically David Blei's 2026 IMS Medallion Lecture program, attempt to resolve this through computational innovations (probabilistic symmetries, simulation-based inference, and calibration studies) and terminological shifts (e.g., "Bayesian empirical Bayes," "population posterior"). The central problem is whether these computational and conceptual extensions resolve the fundamental incoherence of EB—specifically the failure to propagate uncertainty in the prior/hyperparameter—or merely obscure it. The authors contend that the computational cost of maintaining full hierarchical discipline has fallen sufficiently low that the "shortcut" of EB is no longer justified by efficiency gains.

Methodology and Theoretical Framework
The paper employs a theoretical and comparative analysis rather than new experimental simulations. It contrasts three distinct inferential approaches:

1. Full Hierarchical Bayes: Specifying a joint probability model with a proper hyperprior $\rho(\eta)$ on hyperparameters $\eta$, yielding a coherent posterior $p(\theta | x) = \int p(\theta | x, \eta)p(\eta | x)d\eta$.
2. Classical Empirical Bayes (Type II ML): Estimating $\eta$ via marginal maximum likelihood (or method of moments) to obtain $\hat{\eta}(x)$, then computing $p(\theta | x, \hat{\eta})$. This ignores the variability of $\hat{\eta}$.
3. Blei's Modern Program:
   • Bayesian Empirical Bayes (BEB): Uses probabilistic symmetries (exchangeability, shift invariance) and ergodic decompositions to define a directing measure $g$, estimated via variational inference or neural networks.
   • Population Posterior: Defines the target as $q^*(\theta) = E_{X \sim F}[p(\theta | X)]$, an expectation over replicate datasets, rather than the posterior on realized data.
   • Simulation-Based EB: Uses implicit likelihoods and amortized inference networks to approximate the population posterior.

The authors analyze these through the lens of the Tweedie formula ($E[\theta|x] = x + \sigma^2 \nabla \log m(x)$), distinguishing between:

• f-modeling: Smoothing the marginal density $m(x)$ directly. The authors argue this does not guarantee the resulting estimator arises from a valid prior, potentially violating admissibility and coherence.
• g-modeling / NPMLE: Estimating a mixing distribution $\pi$. While coherent, the resulting prior is discrete with finite support, which may fail to capture heavy tails required for sparse signal recovery.
• Horseshoe Hierarchical Bayes: A continuous global-local shrinkage prior (half-Cauchy hyperprior) that yields a valid Tweedie formula and minimax optimality in sparse normal means problems.

Key Contributions and Arguments
The paper makes four primary technical arguments:

1. The "Bayesian Empirical Bayes" Distinction: The authors clarify that Blei's "Bayesian empirical Bayes" (BEB) is structurally distinct from the Deely–Lindley "Bayes empirical Bayes." Deely–Lindley place a proper hyperprior on a finite-dimensional parameter. Blei's BEB fits an infinite-dimensional directing measure via maximum marginal likelihood (MML) using neural networks as optimization tools, without placing a posterior over the directing measure. Consequently, BEB inherits the data-dependence and undercoverage issues of classical EB, merely shifting the plug-in estimation to a higher level of abstraction.

2. The Nature of the "Population Posterior": The paper argues that the "population posterior" ($q^*$) is a frequentist object (an expectation over hypothetical replicates) rather than a Bayesian posterior conditional on realized data. While it has valid predictive interpretations (linked to PAC-Bayesian theory), marketing it as a "posterior" creates confusion regarding uncertainty quantification. The authors propose renaming it the "population predictive distribution" to reflect its frequentist averaging nature.

3. Critique of f-modeling and the Tweedie Formula: The authors refine the Datta–Polson critique of Efron's f-modeling. They demonstrate that a smoothed marginal score function does not necessarily correspond to a valid prior. Therefore, the resulting "posterior mean" is not a genuine Bayes rule and lacks the admissibility and minimax guarantees associated with proper priors. In contrast, the horseshoe Tweedie formula is derived from a proper hierarchical prior (half-Cauchy scale mixture), ensuring the shrinkage profile is a genuine Bayes rule with optimal tail behavior for sparse signals.

4. Simulation-Based Inference (SBI) and Targets: The paper distinguishes between SBI methods that target the standard posterior $p(\theta|x)$ (e.g., Neural Posterior Estimation, Generative Bayesian Computation) and Blei's SBI-EB, which targets the population posterior. The authors argue that modern SBI tools are capable of computing the full hierarchical posterior directly; thus, using them to approximate a population-averaged object is a choice of target, not a computational necessity.

Results and Findings

• Coherence and Uncertainty: The paper establishes that EB shortcuts (plug-in estimators) systematically underestimate uncertainty, leading to undercoverage in credible intervals, particularly in small-sample or sparse regimes. Full hierarchical Bayes propagates hyperparameter uncertainty correctly.
• Optimality: In the sparse normal means problem, the horseshoe hierarchical Bayes estimator achieves minimax optimal rates. The authors show that NPMLE (a coherent EB method) produces a discrete prior that cannot match the heavy-tailed behavior of the horseshoe, leading to suboptimal performance in tail inference.
• Computational Trade-offs: The authors assert that with modern variational inference, neural amortization, and simulation-based methods, the computational cost of full hierarchical Bayes is now negligible compared to the inferential cost of the EB shortcut.
• Case Studies:
  • Pharmacovigilance (DuMouchel): Demonstrates that successful borrowing of strength relies on a real hierarchy (Gamma-Poisson), where the EB step is a computational expedient, not a foundational necessity.
  • Calibration Studies: Shows that combining experimental and observational data via calibration requires a hierarchical model for bias. Plugging in MLE estimates for bias parameters fails to propagate uncertainty, whereas a hierarchical model with a horseshoe prior on bias scales provides coherent inference.

Significance and Recommendations
The paper concludes that the "renaming" of empirical Bayes concepts (e.g., to "Bayesian empirical Bayes") does not resolve the underlying statistical incoherence identified by Lindley, Good, and DuMouchel. The significance of the work lies in its call to redeploy the impressive computational machinery of modern statistics (variational inference, neural networks, SBI) in service of properly hierarchical Bayes rather than as a substitute for it.

The authors recommend:

1. Adopting global-local shrinkage priors (specifically the horseshoe and its variants) as the default for sparse problems, as they provide coherent uncertainty quantification and minimax optimality.
2. Treating the "population posterior" as a frequentist predictive object rather than a Bayesian posterior.
3. Recognizing that the computational gap between EB and full hierarchical Bayes has closed, removing the primary justification for the EB shortcut.
4. Ensuring that any use of "empirical" methods is explicitly framed as an approximation to a hierarchical model, with clear acknowledgment of the conditions under which the approximation fails (e.g., few groups, high sparsity).

The paper does not claim that empirical Bayes is useless, but rather that its modern computational extensions should be viewed as tools to compute full hierarchical posteriors, thereby preserving the discipline of Bayesian inference while achieving the computational efficiency once thought to require the EB shortcut.