📊 statistics

Bayesian Empirical Bayes: Simultaneous Inference from Probabilistic Symmetries

This paper introduces Bayesian Empirical Bayes (BEB), a generalized framework for simultaneous inference that extends classical methods by leveraging probabilistic symmetries and ergodic decompositions to handle complex structures like arrays, covariates, and spatial processes, supported by scalable variational and neural network algorithms.

Original authors: Bohan Wu, Eli N. Weinstein, David M. Blei

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bohan Wu, Eli N. Weinstein, David M. Blei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of statistics, researchers often face a problem of too much data and too little context. Imagine a scientist trying to understand the health of a single patient, but they only have a noisy, blurry measurement. If they look at that patient in isolation, the answer might be wrong. But if they look at thousands of similar patients, patterns emerge. This is the heart of "empirical Bayes," a method that allows researchers to learn from the collective experience of a group to make better guesses about individuals. It works by assuming that everyone in the group shares a common underlying rule, or "prior," which can be discovered by studying the group as a whole. Once that rule is found, it acts as a guide, helping to clean up the noisy data for every single person in the set.

For decades, this powerful technique relied on a strict assumption: that the people or items being studied were all independent and identical, like a bag of mixed marbles where every marble is just as likely to be red or blue as any other. This assumption made the math work, but it often failed in the real world. Modern data is rarely so simple. Genes in a body are arranged in complex networks, air quality sensors are placed across a city with specific geography, and brain connections form intricate maps. In these cases, the "marbles" are not independent; they are arranged in rows and columns, or spread across space and time, influencing each other in structured ways. When the old rules of independence were applied to these complex structures, the results were often poor, leaving the noise uncleaned and the patterns hidden.

A team of researchers at Columbia University and the Technical University of Denmark has now developed a new way to apply this logic to structured data. They call their approach "Bayesian Empirical Bayes." Instead of forcing complex data into a box of simple, independent items, they ask a different question: what symmetries does this data possess? In everyday terms, a symmetry is a way of rearranging the data that leaves its essential nature unchanged. For example, if you swap the names of two patients in a medical study, the overall pattern of disease might stay the same. If you shift a map of air quality one block to the east, the general trends might remain identical. The researchers realized that these symmetries are the key. They proved that for almost any kind of structured data—whether it is a grid of gene activity, a map of brain connections, or a timeline of weather readings—there is a mathematical way to describe the hidden rules that govern the data, based entirely on these symmetries.

The team built a new method that uses these symmetries to find the hidden rules. They treat the unknown rule not as a fixed number, but as a flexible shape that can be learned from the data itself. By assuming that the data respects a specific symmetry, such as the ability to swap rows and columns in a matrix without changing the story the data tells, they can derive a new kind of statistical model. This model allows them to estimate the hidden, true values behind the noise. To make this work on massive datasets, they paired their theory with modern tools from artificial intelligence, using neural networks to learn the complex shapes of these hidden rules and variational inference to solve the heavy calculations quickly.

To test their idea, the researchers applied it to several real-world challenges. They started with a simple task: cleaning up a noisy image of a handwritten digit. When they treated the image as a simple list of independent pixels, the result was decent. But when they treated it as a grid with its own row and column symmetries, the image became much clearer, revealing the digit with far greater accuracy. They then moved to more complex biological data, looking at gene expression in breast cancer patients. Here, the new method successfully separated the true biological signals from the background noise, outperforming existing techniques that had been the standard for years. They also applied it to a map of the human brain, where connections between different regions form a symmetric network, and to air quality data from New York City, where measurements are taken at specific locations over time. In every case, the new method was able to borrow strength from the structure of the data, using the relationships between neighbors to fill in gaps and smooth out errors.

The results were consistent across simulations and real-world tests. In computer experiments where the true answer was known, the new method reduced errors significantly compared to older approaches, especially when the data was very noisy. In the New York City air quality study, the method was able to fill in missing weeks of data and smooth out erratic spikes, providing a clearer picture of how pollution moved through the city. It even identified distinct patterns, showing that air quality in Manhattan behaved differently from that in Queens, a nuance that simpler methods missed. The researchers found that the more complex the structure of the data, the more valuable their symmetry-based approach became.

This work does not claim to solve every statistical problem, nor does it suggest that the old methods are useless. Rather, it offers a principled way to extend the power of learning from the group to the messy, structured reality of modern science. By recognizing that data often comes with built-in symmetries, the researchers have provided a new toolkit for scientists to uncover hidden truths in gene maps, brain networks, and environmental sensors. The method suggests that when we stop trying to force data into a simple, independent mold and instead respect the natural symmetries of the world, we can see the signal much more clearly.

Technical Summary: Bayesian Empirical Bayes: Simultaneous Inference from Probabilistic Symmetries

1. Problem Statement

Classical Empirical Bayes (EB) methods, pioneered by Robbins (1956) and Efron (2012), improve simultaneous inference by "learning from the experience of others." These methods typically assume that latent variables $z_1, \dots, z_n$ are independent and identically distributed (i.i.d.) draws from an unknown prior $g$ . This i.i.d. assumption is justified by the exchangeability of the sequence, which, via de Finetti's representation theorem, allows the joint distribution to be viewed as a mixture of i.i.d. laws.

However, modern data applications frequently violate the i.i.d. assumption. Data often possess complex structures, such as:

Arrays and Graphs: Gene expression matrices or social networks where rows and columns exhibit dependencies.
Spatial and Temporal Processes: Air quality measurements or time series where observations are correlated by location or time.
Covariate-Dependent Data: Hypotheses where the prior distribution depends on known covariates.

In these settings, the latent variables are not exchangeable in the classical sense, rendering standard EB methods (which rely on a single global prior $g$ ) suboptimal or inapplicable. The paper addresses the challenge of extending EB to these non-i.i.d. settings by leveraging probabilistic symmetries other than simple exchangeability.

2. Methodology: Bayesian Empirical Bayes (BEB)

The authors propose Bayesian Empirical Bayes (BEB), a framework that generalizes EB by replacing the assumption of exchangeability with general probabilistic symmetries. The core insight is that while the latent variables may not be i.i.d., their joint distribution often exhibits invariance under specific group actions (e.g., row/column permutations, spatial shifts).

Theoretical Foundation

The methodology rests on the Ergodic Decomposition Theorem (Varadarajan, 1963; Kallenberg, 2005), which generalizes de Finetti's theorem.

Symmetry Assumption: Let $\Phi$ be a group of transformations acting on the index set $\Omega$ (e.g., permutations for arrays, shifts for spatial data). The prior $p(z)$ is assumed to be $\Phi$ -invariant.
Ergodic Decomposition: Any $\Phi$ -invariant distribution can be uniquely decomposed into a mixture of $\Phi$ -ergodic distributions:
$p(z) = \int_{\mathcal{G}} p(z | g) d\mu(g)$
Here, $g$ is an invariant variable (or directing random measure) that indexes the ergodic family $\mathcal{E} = \{p(z|g)\}$ .
BEB Formulation:
1. Model: The data $x$ is generated from latent variables $z$ via a known likelihood $p(x|z)$ . The prior on $z$ is $\Phi$ -invariant, implying the hierarchical structure:
  $g \sim \mu, \quad z | g \sim p(z|g), \quad x | z \sim p(x|z)$
2. Estimation: Instead of integrating over the mixing distribution $\mu$ (as in full Bayesian inference), BEB estimates the invariant variable $g$ by maximizing the marginal likelihood (Maximum Marginal Likelihood Estimation, MMLE):
  $\hat{g} \in \arg\max_{g} \log \int p(x|z) p(z|g) dz$
3. Inference: The posterior for the latent variables is approximated by plugging in the point estimate $\hat{g}$ :
  $\hat{p}(z|x) \approx p(z | x, \hat{g})$

Algorithmic Implementation

The paper develops scalable algorithms based on Variational Inference (VI) and Neural Networks to handle the intractable integrals in the marginal likelihood:

Matrix Recovery (Separate/Joint Exchangeability): For array data, the invariant variable $g$ is a measurable function on $[0,1]^3$ (per the Aldous-Hoover theorem). The authors parameterize this function using a neural network and optimize the Evidence Lower Bound (ELBO) using a structured variational family that factorizes row and column latent factors.
Spatial Regression (Shift Invariance): For spatial data, the invariant variable is the spectral density of a stationary Gaussian process. The authors model the spectral density as a mixture of Gaussians and estimate its parameters via marginal likelihood maximization, leveraging Bochner's theorem to link spectral density to the covariance kernel.
Covariate-Informed EB: The framework extends to cases where the invariant function depends on observed covariates, allowing for conditional priors.

3. Key Contributions

Generalization of EB: The paper provides a unified framework for EB that moves beyond i.i.d. sequences to general probabilistic symmetries (arrays, spatial processes, covariate-dependent structures).
Theoretical Unification: It establishes that classical EB is a special case of BEB where the symmetry is exchangeability and the ergodic decomposition yields i.i.d. laws. It formalizes the connection between ergodic theory and empirical Bayes.
New Inference Procedures:
- EBMR (Empirical Bayes Matrix Recovery): New methods for denoising matrices under separate and joint exchangeability, parameterized by neural networks.
- Spatial EB: A new approach to spatial regression that learns the spectral density of the latent process directly from data.
- Covariate-Informed EB: Methods for incorporating side information into the prior structure.
Scalable Algorithms: The development of variational inference algorithms that approximate the nonparametric maximum likelihood for these complex, high-dimensional models.

4. Results

The authors evaluate BEB through simulations and real-world datasets:

Simulations (Matrix Recovery): In 60 simulated settings with varying noise levels and data-generating mechanisms (linear, sinusoidal, etc.), EBMR with a separately exchangeable prior consistently outperformed Nonparametric Maximum Likelihood Estimation (NPMLE) and Empirical Bayes Matrix Factorization (EBMF). The performance gap widened with higher noise levels, demonstrating the "blessing of noise" where borrowing strength across the matrix structure becomes more critical.
Simulations (Spatial Regression): EB spatial regression significantly outperformed regression-adjusted NPMLE. While NPMLE error plateaued as sample size increased, the MSE of the spatial EB estimator decreased sharply, indicating that shift invariance is a more informative symmetry for spatial data than full exchangeability.
Real Data Applications:
- Gene Expression & Brain Connectome: Applied to a 251x226 cancer gene-expression matrix and a 246x246 brain connectivity matrix. EBMR methods achieved lower relative Mean Squared Error (MSE) than NPMLE, EBMF, and singular-value thresholding baselines, particularly at high noise levels.
- NYC Air Quality: Applied to PM2.5 data from 4 monitoring stations. The method successfully denoised and interpolated missing data, capturing bimodal spatio-temporal regimes (Manhattan vs. Queens) that standard Gaussian Process baselines with fixed kernels failed to model. The fitted spectral density revealed distinct temporal cycles and spatial scales.

5. Significance and Claims

The paper claims that Bayesian Empirical Bayes offers a principled way to extend the power of empirical Bayes to structured data. By treating the unknown prior as an object of inference encoded by an invariant variable $g$ , BEB allows for simultaneous inference in settings where classical i.i.d. assumptions fail.

Flexibility vs. Efficiency: The authors note a trade-off: weaker symmetry assumptions (e.g., joint vs. separate exchangeability) enlarge the class of admissible priors and increase flexibility but require more data and computational resources.
Modeling Choice: The paper emphasizes that assuming a symmetry is a modeling choice rather than a strict requirement of the data-generating process. Even if the prior is not exactly invariant, positing a symmetry restricts inference to equivariant decision rules, which can be statistically efficient.
Future Directions: The authors identify theoretical analysis (identifiability, consistency, sample complexity) and extensions to more complex symmetry groups (e.g., physical constraints, geometric deep learning) as open areas for future research. They also suggest the need for further sensitivity analysis under model misspecification.

In summary, the paper demonstrates that by leveraging the ergodic decomposition of probabilistic symmetries, one can construct robust, scalable, and accurate simultaneous inference procedures for complex, structured data, outperforming existing denoising and regression methods in both synthetic and real-world scenarios.