📊 statistics

Bayesian factorization via $L_{1/2}$ shrinkage

This paper proposes a Bayesian factor model utilizing an $L_{1/2}$ shrinkage prior to achieve increasing shrinkage with a simple structure that enables efficient exact posterior inference via Gibbs sampling and faster variational approximation, outperforming existing methods in both accuracy and computational efficiency.

Original authors: Shicheng Liu, Qingping Zhou, Yanan Fan, Xiongwen Ke

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shicheng Liu, Qingping Zhou, Yanan Fan, Xiongwen Ke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a massive, chaotic library containing millions of books (data points). Each book has thousands of pages (variables), but you suspect that the story isn't actually that complicated. You think there are only a few core themes (latent factors) driving the plot, and most of the pages are just noise or repetition.

This is the problem of Factor Analysis. It's a way to compress huge amounts of data into a few simple, understandable "themes."

However, there's a catch: How many themes are there? Is it 3? 5? 50? And which pages actually belong to which theme?

The Old Way: The "Heavy Backpack" Problem

For a long time, statisticians used a method called Bayesian Factor Analysis to solve this. They used a special "shrinkage" tool (a prior) to automatically push unimportant themes toward zero, effectively deleting them.

Think of this like a hiker trying to carry a backpack with infinite pockets. The old tools (like the Multiplicative Gamma Process) were like a backpack with a very complex, tangled web of straps.

The Good: It worked well at finding the right number of themes.
The Bad: The straps were so complicated that calculating the answer took forever. It was like trying to untangle a knot while running a marathon. The computer would get tired (slow) and run out of memory.

The New Solution: The "Magic L1/2 Shrinkage"

The authors of this paper, Liu, Zhou, Fan, and Ke, introduced a new tool: the $L_{1/2}$ Shrinkage Prior.

Imagine instead of a tangled web of straps, you have a smart, self-cleaning backpack.

The "Increasing" Rule: This backpack has a special rule: "The further down the list of pockets you go, the tighter the strap gets."
- Pocket #1 (the most important theme) has a loose strap.
- Pocket #10 has a tighter strap.
- Pocket #100 has a strap so tight it crushes the contents to zero.
- Result: The computer naturally ignores the useless, distant pockets without you having to tell it which ones to delete.
Simplicity: Unlike the old tangled straps, this new backpack is simple. It's easy to open, easy to close, and easy to calculate with.

Two Ways to Use the New Backpack

The paper offers two ways to use this new tool, depending on how much time you have:

1. The "Gold Standard" Method (Gibbs Sampler)

What it is: This is like carefully reading every single page of the library to find the perfect themes.
Pros: It is exact. It finds the mathematically perfect answer.
Cons: It is slow. If your library is huge, it might take days.
The Twist: Because the new backpack is so simple, this "slow" method is actually much faster than the old "tangled strap" method. It's like switching from a horse-drawn carriage to a modern car; you're still driving, but you're much faster.

2. The "Speed Demon" Method (Variational Inference)

What it is: This is like skimming the library. You don't read every word; you look at the chapter titles and summaries to get a "good enough" answer very quickly.
Pros: It is blazing fast. It can handle massive datasets (like millions of genes) in seconds.
Cons: It's an approximation, not the exact truth. But, the paper shows it's accurate enough for almost everything.
The Metaphor: If the Gold Standard is a high-resolution photo, this is a slightly compressed JPEG. You can't see the dust on the lens, but you can clearly see the face.

Why Does This Matter? (Real World Examples)

The authors tested their new backpack on two real-life scenarios:

Lung Cancer Genes: They looked at data from 56 patients with 5,000 genes each.
- The Goal: Find which genes cause cancer and group the patients into the right tumor types.
- The Result: Their method successfully separated the different tumor types and found the key genes, doing it more efficiently than the old methods.
Single-Cell Biology (PBMC): They looked at 2,700 individual cells with 32,000 genes each. This is a huge dataset.
- The Goal: Sort the cells into different types (like T-cells, B-cells, etc.).
- The Result: Their "Speed Demon" method (Variational Inference) grouped the cells perfectly, identifying the same cell types as the experts, but in a fraction of the time.

The Bottom Line

This paper is about simplifying the complex.

The authors took a difficult statistical problem (figuring out how many hidden themes exist in massive data) and replaced a complicated, slow machine with a simple, efficient one.

Old Way: Complex, slow, hard to use.
New Way: Simple, fast, and just as accurate.

They essentially gave statisticians a "smart backpack" that automatically knows which items to throw away, allowing us to analyze massive datasets (like entire genomes) without needing a supercomputer.

1. Problem Statement

Factor models are widely used for dimension reduction in high-dimensional data (e.g., genomics, economics). However, two major challenges persist:

Unknown Latent Dimension: The number of latent factors ( $K$ ) is typically unknown.
Inconsistent Estimation: Traditional methods like Principal Component Analysis (PCA) often fail to consistently estimate eigenvectors in high-dimensional settings ( $p \gg n$ ).
Computational Complexity: Existing Bayesian approaches that address these issues (e.g., Multiplicative Gamma Process (MGP), Spike-and-Slab priors) often rely on complex hierarchical structures. This complexity complicates posterior inference, making Markov Chain Monte Carlo (MCMC) sampling slow and Variational Inference (VI) difficult to derive due to non-conjugacy or intractable posteriors.

The authors aim to develop a Bayesian factor model that achieves increasing shrinkage (where loadings for higher-indexed factors are shrunk more aggressively toward zero) while maintaining a simple prior structure to facilitate efficient exact (Gibbs) and approximate (Variational) inference.

2. Methodology

2.1 The $L_{1/2}$ Shrinkage Prior

The core innovation is the application of the $L_{1/2}$ shrinkage prior (originally proposed by Ke & Fan, 2025) to the factor loading matrix $B$ .

Structure: The prior is placed on individual elements $B_{jk}$ of the loading matrix.
Global Shrinkage: A global shrinkage parameter $\lambda_k$ is assigned to the $k$ -th column. Crucially, $\lambda_k$ follows a Gamma distribution where the shape parameter increases with $k$ ( $a + kc_1$ ). This ensures that as the column index increases, the global shrinkage parameter increases stochastically, forcing elements in later columns toward zero.
Local Shrinkage: The prior utilizes a Gaussian mixture representation involving local shrinkage parameters ( $\tau^2_{jk}$ ) and auxiliary variables ( $v_{jk}$ ) to handle the non-Gaussian $L_{1/2}$ penalty.
Theoretical Properties: The authors prove that under specific hyperparameter constraints ( $a \ge 4, c_1 + c_2 > 1/4$ $a \geq 4, c_{1} + c_{2} > 1/4$ ):
- The prior has full support on the space of valid loading matrices.
- The increasing shrinkage property holds in probability (a stronger condition than MGP, which only holds in expectation).
- The posterior is weakly consistent for the covariance matrix.

2.2 Exact Inference: Gibbs Sampler

Due to the specific hierarchical structure of the $L_{1/2}$ prior, the authors derive a Gibbs sampler with closed-form conditional posteriors for all parameters ( $B, \Omega, \eta, \lambda, \tau^2$ ).

Key Advantage: Unlike MGP or Spike-and-Slab priors, the conditional posteriors of global and local shrinkage parameters are independent across different latent dimensions $k$ . This simplifies the sampling steps significantly.
Algorithm: The sampler iteratively updates the error variance, factor loadings (via multivariate normal), latent factors, and shrinkage parameters (via Gamma and Inverse Gaussian distributions).

2.3 Approximate Inference: Collapsed Variational Inference (VI)

To address the computational cost of MCMC for large datasets, a novel Collapsed Variational Inference algorithm is proposed.

Collapsing: Global and local shrinkage parameters are integrated out (marginalized) to improve approximation accuracy.
Variational Family:
- $q(\eta)$ and $q(\sigma^2)$ are assumed to follow their conditional posterior forms (Normal and Inverse Gamma).
- $q(B)$ is approximated by a Multivariate t-distribution ( $t_\nu$ ). This is chosen because the $L_{1/2}$ prior induces heavy tails, which a Gaussian approximation might miss.
Optimization:
- The Evidence Lower Bound (ELBO) is maximized using coordinate ascent.
- Natural Gradient Descent: To optimize the location and scale parameters of the t-distribution, the authors use natural gradient descent (utilizing the Fisher Information matrix) to ensure the positive definiteness of the precision matrix and faster convergence.
- Mirror Descent: The degrees of freedom parameter ( $\nu$ ) is optimized using mirror descent (specifically exponentiated gradient) to handle the constraint $\nu > 2$ .
- Gradient Estimation: Since the prior is non-differentiable at zero, the authors use a surrogate lower bound (Minorize-Maximize approach) and generalized reparameterization gradients (Naesseth et al., 2017) to estimate gradients for the shape parameter of the Gamma distribution.

3. Key Contributions

Novel Prior Specification: Introduction of the $L_{1/2}$ shrinkage prior for factor models, which guarantees increasing shrinkage in probability and offers a stronger shrinkage effect than MGP.
Simplified Inference: The prior structure allows for a simple Gibbs sampler with independent conditional posteriors for shrinkage parameters, a feature lacking in MGP and CSP priors.
Advanced Variational Algorithm: Development of a collapsed VI algorithm using Multivariate t-distributions and natural/mirror gradient descent to handle the non-conjugate and non-smooth nature of the $L_{1/2}$ prior.
Scalability: The proposed methods are shown to be highly scalable, particularly when leveraging GPU acceleration for the Gibbs sampler.

4. Results

4.1 Simulation Studies

The authors compared their methods ( $L_{1/2}$ GS and VI) against MGP (GS/VI), CSP (GS), and SSL-IBP (PXL-EM).

Accuracy: The $L_{1/2}$ methods consistently outperformed MGP and CSP in terms of Frobenius norm error and False Discovery Rate (FDR) in estimating the covariance matrix. They achieved accuracy comparable to or better than SSL-IBP in many scenarios, while SSL-IBP sometimes overestimated sparsity (high False Negative Rate).
Dimension Selection: All $L_{1/2}$ methods correctly identified the true number of latent factors ( $K=5$ ) across various data dimensions ( $n, p$ ).
Computational Efficiency:
- VI: The $L_{1/2}$ VI was significantly faster than the Gibbs sampler across all platforms.
- GPU Scaling: On GPUs, the $L_{1/2}$ Gibbs sampler became the fastest among all MCMC methods, outperforming MGP and CSP, which scaled less efficiently.
- Trade-off: The VI algorithm offered the best speed-accuracy trade-off for large-scale problems.

4.2 Real Data Applications

Lung Cancer Dataset ( $n=56, p=5000$ ): The methods successfully identified biologically meaningful latent factors separating four tumor types. The $L_{1/2}$ prior effectively shrunk irrelevant factors, and the Gibbs sampler provided slightly sparser and more accurate loading estimates than VI in this small-sample regime.
PBMC scRNA-seq Dataset ( $n=2700, p=32738$ ): The methods reduced dimensionality to cluster cells. Both GS and VI produced nearly identical clustering results (9 clusters, refined to 7 biologically distinct types) and identified consistent marker genes (e.g., LTB, IL7R for T cells). The $L_{1/2}$ prior successfully shrunk the loading matrix columns toward zero as the index increased, unlike some competitors.

5. Significance and Conclusion

This paper presents a robust framework for Bayesian factor analysis that balances statistical rigor (strong theoretical shrinkage properties) with computational feasibility (efficient exact and approximate inference).

Theoretical Impact: It demonstrates that complex hierarchical priors (like MGP) are not strictly necessary for increasing shrinkage; a carefully parameterized $L_{1/2}$ prior can achieve superior shrinkage properties with a simpler structure.
Practical Impact: The proposed algorithms are highly scalable. The ability to run the Gibbs sampler efficiently on GPUs makes it viable for modern high-dimensional datasets (e.g., genomics) where MCMC was previously too slow. The VI algorithm provides a fast alternative for massive datasets.
Future Directions: The authors suggest extending this increasing shrinkage prior to infinite-width neural networks, though this would require new inference algorithms due to the lack of conjugacy in deep learning contexts.

In summary, the $L_{1/2}$ shrinkage prior offers a compelling alternative to existing Bayesian factor models, providing a unique combination of strong theoretical guarantees, accurate dimension selection, and high computational efficiency.