📊 statistics

Overfitted high-dimensional matrix factorizations via adaptive spectral shrinkage

The paper introduces \texttt{EigenBayes}, a computationally efficient, MCMC-free method for high-dimensional factor analysis that uses adaptive spectral shrinkage and empirical Bayes calibration to automatically determine latent dimensions, provide valid uncertainty quantification, and outperform existing approaches in both simulations and genomics applications.

Original authors: Lorenzo Mauri, David B. Dunson

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lorenzo Mauri, David B. Dunson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a massive, chaotic orchestra playing a symphony. You have thousands of microphones (data points) recording the sound, but you suspect that the music isn't actually made of thousands of independent instruments. Instead, you believe it's driven by a small number of "hidden conductors" (latent factors) leading a few sections, while the rest of the noise is just individual musicians playing slightly out of tune (residual error).

The challenge is: How many conductors are there? And how do you separate the true music from the noise without getting confused?

This paper introduces a new tool called EigenBayes to solve this problem. Here is how it works, broken down into simple concepts:

1. The Problem: Guessing the Number of Conductors

In high-dimensional data (like gene expression or financial markets), we often try to find these hidden patterns.

The Old Way: Traditional methods try to guess the exact number of hidden factors. If they guess too low, they miss important signals. If they guess too high, they get confused by noise.
The "Overfitted" Way: Some modern methods say, "Let's just assume there are way more conductors than we think there are." Then, they use a special "shrinkage" filter to turn down the volume of the fake conductors until they are silent.
- The Catch: These methods are like trying to conduct an orchestra by listening to every single instrument individually. It's incredibly accurate, but it takes a huge amount of time and computing power (like waiting hours for a computer to finish a calculation).

2. The Solution: EigenBayes (The Fast, Smart Filter)

The authors propose EigenBayes, a method that combines the speed of a quick glance with the accuracy of a deep analysis.

Step 1: The Quick Scan (Spectral Estimation):
First, EigenBayes does a very fast mathematical "scan" of the data (using something called Singular Value Decomposition). It assumes there are more conductors than there actually are (an "overfitted" rank). Think of this as turning on all the lights in the orchestra hall to see everyone at once.
Step 2: The Smart Shrinkage (Adaptive Filtering):
Instead of running a slow, complex simulation to figure out which conductors are real, EigenBayes uses a clever "Empirical Bayes" trick. It looks at the data and automatically adjusts the volume knobs for each conductor.
- If a conductor is playing a loud, clear note (strong signal), the volume stays up.
- If a conductor is just making random noise (weak signal), the volume is turned all the way down to zero.
- The Magic: It does this differently for every "song" (outcome) and every "conductor" (latent dimension). It adapts to the specific noise level of each part of the data.
Step 3: The Result:
Because it uses this smart filtering instead of a slow simulation, the math "factors" (splits) neatly. This means the computer can solve thousands of tiny problems in parallel, making it massively faster than previous methods. It can process data in seconds that used to take hours.

3. Why It's Better

The paper claims three main advantages:

Speed: It bypasses the need for slow, repetitive computer sampling (Markov chain Monte Carlo). It's like getting a high-definition photo instantly instead of waiting for a printer to build it pixel by pixel.
Automatic Tuning: You don't need to be a genius to set the parameters. The method figures out the right amount of "shrinkage" on its own based on the data. If you guess the number of conductors is too high, the method automatically silences the extras.
Honesty (Uncertainty Quantification): Unlike some fast methods that just give you a single "best guess," EigenBayes tells you how confident it is. It provides a range of likely answers (confidence intervals) that are statistically valid, so you know when the data is too noisy to be sure.

4. Real-World Test: The Gene Orchestra

The authors tested this on a real dataset involving gene expression (628 samples of immune cells and 5,000 genes).

They compared EigenBayes against the best existing methods.
The Result: EigenBayes was just as accurate (or better) at predicting the patterns in the genes, but it was thousands of times faster. While other methods took about 30 minutes to run a single test, EigenBayes took less than one second.

Summary

Think of EigenBayes as a super-smart, ultra-fast audio engineer. Instead of spending hours manually adjusting every fader on a mixing board, it uses a smart algorithm to instantly identify the main melodies, mute the background noise, and tell you exactly how loud the music is, all in a fraction of a second. It allows scientists to analyze massive datasets without needing a supercomputer or waiting days for results.

Technical Summary: Overfitted high-dimensional matrix factorizations via adaptive spectral shrinkage

Problem Statement

Estimating covariance matrices in high-dimensional settings (where the ambient dimension $p$ is large, often exceeding the sample size $n$ ) is a fundamental challenge in statistics. Standard factor models decompose the covariance matrix $\Theta$ into a low-rank component ( $\Lambda\Lambda^\top$ ) and a diagonal residual component ( $\Sigma$ ). A critical difficulty lies in selecting the latent dimension $k$ . This selection is particularly challenging when the factor model holds only approximately or in low signal-to-noise ratio (SNR) scenarios.

Existing approaches face a trade-off:

Spectral methods (e.g., PCA-based estimators) are computationally efficient but often suffer from poor estimation accuracy when latent components are not significantly larger than the noise, and they lack formal uncertainty quantification.
Bayesian overfitted factor models specify an upper bound $H$ on the latent dimension and use structured shrinkage priors to remove unnecessary factors. While effective, these methods typically rely on computationally expensive Markov chain Monte Carlo (MCMC) sampling (e.g., reversible jump MCMC or Gibbs samplers), which can suffer from slow convergence and mixing in high dimensions. Variational approximations offer speed but often underestimate uncertainty.
Hybrid approaches that combine spectral pre-estimation with Bayesian inference exist but often lack mechanisms for adaptive rank selection or adaptation to varying SNRs across outcomes.

Methodology: EigenBayes

The authors propose EigenBayes, a scalable approach that combines the computational efficiency of spectral estimators with the uncertainty quantification and adaptive shrinkage of Bayesian methods. The core methodology proceeds as follows:

Spectral Initialization: The method begins by estimating latent factors using an overestimated latent dimension $H$ (where $H \geq k$ ). The initial estimate of the latent factor matrix $\hat{M}$ is derived from the leading left singular vectors of the data matrix $Y$ , scaled by $\sqrt{n}$ .
Surrogate Bayesian Regression: Instead of sampling the full joint posterior, the method treats the latent factors as fixed (conditioned on $\hat{M}$ $\hat{M}$ ) and infers the loading matrix $\Lambda$ $Λ$ and residual variances $\Sigma$ $Σ$ via $p$ $p$ independent surrogate Bayesian linear regression tasks.
- The model for the $j$ -th outcome is $y^{(j)} = \hat{M}\lambda_j + \epsilon_j$ .
- A conjugate Normal-Inverse Gamma prior is placed on the loadings $\lambda_j$ and variance $\sigma_j^2$ .
- Crucially, the prior includes hyperparameters $\tau_j^2$ (outcome-specific) and $\psi_l^2$ (component-specific) to control shrinkage.
Adaptive Hyperparameter Calibration (Empirical Bayes): The hyperparameters are tuned via an Empirical Bayes moment-matching strategy. The procedure calibrates the prior expected magnitudes of the loadings to their spectral estimates while correcting for the overestimation of the latent dimension.
- The noise level is estimated using the $(H+1)$ -th singular value ( $d_{H+1}$ ), which represents the bulk noise in overfitted components.
- The shrinkage parameters $\hat{\psi}_l^2$ are set proportional to the relative contribution of variability explained by the $l$ -th component after subtracting the noise floor.
- The parameters $\hat{\tau}_j^2$ adapt to the signal-to-noise ratio of each outcome.
Posterior Factorization and Uncertainty Quantification: Conditioning on the spectral estimate causes the posterior distribution to factorize across outcomes, reducing inference to $p$ $p$ independent regressions. This enables embarrassingly parallel computation.
- To ensure valid frequentist coverage, a posterior variance inflation factor $\rho > 1$ is applied, derived from the work of Chattopadhyay et al. (2024), to correct for slight undercoverage in the raw posterior.
- The resulting posterior allows for full uncertainty quantification of the covariance entries and latent factors without MCMC.

Key Contributions

Computational Efficiency: EigenBayes bypasses MCMC by utilizing a surrogate regression framework, making it significantly faster than overfitted Bayesian factor models (e.g., MGSP) while retaining full uncertainty quantification.
Adaptive Rank Selection: The method automatically shrinks superfluous latent components (those beyond the true rank $k$ ) to zero. Theoretical results show that the empirical Bayes estimates of the shrinkage parameters for redundant components converge to zero at a fast rate, effectively adapting to the true rank without explicit model selection.
Heterogeneity Adaptation: The approach adapts to varying signal-to-noise ratios across different outcomes (variables) and latent dimensions, providing stronger shrinkage for noisy variables and weaker shrinkage for informative ones.
Theoretical Guarantees: The paper establishes:
- Automatic Rank Adaptation: The prior variance for redundant components vanishes asymptotically.
- Consistency: The point estimator achieves favorable convergence rates for the low-rank and full covariance matrices, behaving asymptotically like an oracle spectral estimator that knows the true rank.
- Uncertainty Quantification: A Central Limit Theorem (CLT) is proven for the point estimator, and a Bernstein-von Mises result characterizes the asymptotic behavior of the induced posterior distribution, providing one of the first formal frequentist justifications for Empirical Bayes inference in overfitted factor models.

Results

Numerical Experiments:
The authors compared EigenBayes against several state-of-the-art alternatives: Barigozzi & Cho (2018), Principal Component (PC) estimators, Rockova & George (2016) (ROTATE), Chattopadhyay et al. (2024) (FABLE), and Bhattacharya & Dunson (2011) (MGSP).

Accuracy: EigenBayes achieved state-of-the-art accuracy in reconstructing the covariance matrix across various scenarios, including dense, heterogeneous, and sparse loading structures. It outperformed competitors, particularly in low sample size settings and when the latent dimension was overestimated.
Robustness: The method was remarkably robust to the choice of the upper bound $H$ . Even when $H$ was set conservatively (e.g., $\lfloor \sqrt{p} \rfloor$ ), EigenBayes performed nearly as well as if the true rank were known, whereas competitors often degraded in performance as $H$ increased.
Uncertainty Quantification: In terms of interval estimation, EigenBayes provided accurate frequentist coverage (close to nominal levels) across covariance entries. In contrast, FABLE suffered from non-negligible undercoverage, and MGSP showed undercoverage in high-dimensional settings.

Genomics Application:
Applied to the ImmGen microarray dataset (628 samples, 5000 genes), EigenBayes demonstrated superior out-of-sample predictive performance (measured by log-likelihood) compared to competitors across a wide range of latent dimensions.

Speed: EigenBayes was orders of magnitude faster than MGSP, with model fitting taking less than 1 second on average compared to approximately 30 minutes for MGSP per replication.

Significance and Claims

The paper claims that EigenBayes offers a compelling alternative for practitioners dealing with high-dimensional factor analysis where the latent dimension is difficult to specify or the signal-to-noise ratio is low. By combining the speed of spectral methods with the adaptivity and uncertainty quantification of Bayesian approaches, it addresses the computational bottlenecks of existing Bayesian overfitted models.

The authors emphasize that their work provides a formal frequentist theoretical justification for Empirical Bayes inference in this context, demonstrating that the method recovers the low-rank structure at favorable rates and provides asymptotically valid uncertainty intervals. The method is presented as a robust solution that reduces the need for extensive tuning of the latent dimension while maintaining high accuracy and computational scalability.

The paper concludes by noting limitations and future directions, such as extending the framework to non-Gaussian data (e.g., count or binary data), localizing shrinkage to individual matrix entries, and handling dependence structures (serial or spatial correlation), but does not claim these extensions are currently realized.