📊 statistics

Rényi's $\alpha$ -divergence variational Bayes for spike-and-slab high-dimensional linear regression

This paper proposes a mean-field variational Bayes framework for sparse high-dimensional linear regression that replaces the standard Kullback-Leibler divergence with Rényi's $\alpha$ -divergence to offer flexible trade-offs between zero-forcing and mass-covering behaviors, deriving efficient CAVI and stochastic variational inference algorithms that demonstrate competitive performance across various sparsity configurations.

Original authors: Chadi Bsila, Yiqi Tang, Kaiwen Wang

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chadi Bsila, Yiqi Tang, Kaiwen Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a massive mystery, but you have a strange problem: you have a list of a thousand suspects, yet you know for a fact that only a handful of them actually committed the crime. In the world of statistics, this is called "sparse high-dimensional regression." It's the challenge of finding the few important signals hidden inside a mountain of noisy data. Usually, statisticians use a method called "Bayesian inference" to solve this, which is like gathering all the clues to build a perfect picture of who did it. However, building that perfect picture is so computationally heavy that it can take computers days or even weeks to crunch the numbers, especially when the suspect list gets huge.

To speed things up, scientists invented a shortcut called "Variational Bayes." Instead of building the perfect picture, they try to find the closest possible simpler picture that is quick to draw. Traditionally, they measure how "close" their sketch is to the real picture using a ruler called the "Kullback–Leibler (KL) divergence." Think of this ruler as a strict teacher who only cares if your sketch matches the most likely parts of the crime scene, often ignoring the weird, less likely details. But what if that strict teacher is too rigid? What if we need a ruler that can be a bit more flexible, sometimes focusing on the most likely suspects (zero-forcing) and other times making sure we don't miss any potential suspects, even the unlikely ones (mass-covering)? This is where a new, more flexible ruler called "Rényi's α-divergence" comes in. It has a dial, labeled $\alpha$ , that lets you change how strict or flexible your search for the truth is.

In this paper, the authors, Chadi Bsila, Yiqi Tang, and Kaiwen Wang, decide to swap out the old, strict ruler for this new, adjustable one to solve the "thousand suspects" problem. They propose two new methods, which they call AlphaVB and AlphaSVB, designed to find the best sketch of the data using this flexible dial. They tested their ideas by creating thousands of fake crime scenes (simulations) where they knew exactly which "suspects" (variables) were guilty. Their goal was to see if turning the $\alpha$ dial could help them find the guilty parties more accurately and estimate their actions better than the old methods.

The results of their simulations show that the new methods are quite promising, but with a catch. The first method, AlphaVB, which uses a step-by-step optimization technique, performed very well. In fact, it was competitive with the best existing methods in the field. The authors found that for AlphaVB, setting the dial to a value just barely above 1 (specifically $\alpha = 1.01$ ) worked the best across most scenarios. This setting allowed the method to be accurate in spotting the guilty variables (high True Positive Rate) while rarely accusing innocent ones (low False Discovery Rate). However, the second method, AlphaSVB, which uses a different, more random sampling approach, struggled. In their simulations, AlphaSVB did not perform as well as the other top methods, often making more mistakes in both identifying the right variables and estimating their values. The authors suggest that while the random sampling idea is conceptually simpler, it might not be the best fit for this specific type of high-dimensional problem.

One of the most interesting discoveries in the paper is how sensitive the results are to the setting of the $\alpha$ dial. For AlphaVB, the authors found that while a small value like 1.01 was generally the best, turning the dial to very high numbers (like 5 or 100) made the method extremely good at not making false accusations (achieving a perfect False Discovery Rate of 0.00 in some cases). However, this came at a steep price: the method became terrible at finding the actual guilty suspects and estimating their values correctly. It was like a detective who is so afraid of accusing an innocent person that they refuse to accuse anyone at all, even the obvious culprits. Conversely, for the struggling AlphaSVB method, the authors found that setting the dial to values less than 1 (like 0.9) actually helped it perform better than when it was set to higher values, though it still lagged behind the other methods.

Ultimately, the paper suggests that using this flexible "Rényi" ruler is a powerful tool, but it requires careful tuning. The authors conclude that their AlphaVB method is a strong, competitive choice for researchers dealing with sparse data, offering a nice balance between speed and accuracy. They also highlight that the "best" setting for the $\alpha$ dial depends entirely on the specific situation: if you need to be absolutely sure you aren't accusing the wrong person, you might crank the dial up high, but if you need to find as many clues as possible, a setting closer to 1 is better. While their random-sampling method (AlphaSVB) didn't win the race in these tests, the work opens the door for future experiments to see if this flexible approach can be improved or applied to other types of statistical puzzles.

Technical Summary: Rényi's $\alpha$ -divergence Variational Bayes for Spike-and-Slab High-Dimensional Linear Regression

Problem Statement
The paper addresses the challenge of sparse high-dimensional linear regression, where the number of predictors $p$ significantly exceeds the number of observations $n$ ( $p \gg n$ ). The primary goals are variable selection and coefficient estimation. While Bayesian methods offer robust uncertainty quantification through posterior distributions, their reliance on Markov Chain Monte Carlo (MCMC) renders them computationally prohibitive in ultra-high-dimensional settings. Variational Inference (VI) offers a faster alternative by converting integration into optimization, typically minimizing the Kullback–Leibler (KL) divergence. However, standard KL-based VI may lack flexibility in capturing the specific behaviors required for sparse models, such as balancing "zero-forcing" (concentrating on high-density regions) and "mass-covering" (capturing broader support).

Methodology
The authors propose two Variational Bayes (VB) algorithms, AlphaVB and AlphaSVB, which replace the standard KL divergence objective with Rényi's $\alpha$ -divergence. This one-parameter generalization ( $\alpha \in (0, \infty) \setminus \{1\}$ ) allows for tunable behavior: $\alpha > 1$ induces zero-forcing behavior, while $\alpha < 1$ induces mass-covering behavior.

The models utilize a Laplace spike-and-slab prior within a mean-field variational family. The prior assumes a hierarchical structure where coefficients are either zero (spike) or drawn from a Laplace distribution (slab). The variational family consists of independent mixtures of a Gaussian slab and a Dirac spike at zero.

Two distinct optimization strategies are derived:

AlphaVB (Coordinate Ascent Variational Inference):
- This algorithm derives coordinate-wise updates for the variational parameters ( $\mu_i, \sigma_i, \gamma_i$ ).
- Because the Rényi objective involves the expectation of a non-linear term (specifically, the $(\alpha-1)$ -th power of the likelihood ratio), exact closed-form updates are intractable.
- To overcome this, the authors employ the multivariate Delta method (a second-order Taylor expansion) to approximate the expectation. This allows for the derivation of update equations that resemble standard CAVI but include correction terms dependent on $\alpha$ .
- For the inclusion probability $\gamma_i$ , the authors utilize Jensen's inequality to derive an update equation that matches the form found in Ray and Szabó (2022) for the KL setting.
AlphaSVB (Stochastic Variational Inference):
- This algorithm optimizes a Variational Rényi (VR) bound, an analogue to the Evidence Lower Bound (ELBO) used in standard VI.
- Instead of deterministic coordinate updates, AlphaSVB uses Monte Carlo (MC) sampling to approximate the expectations in the VR bound.
- It computes gradients of the MC-approximated bound with respect to the variational parameters using importance weights derived from the Rényi divergence formulation.
- The parameters are updated via stochastic gradient ascent.

Key Contributions

Formulation: The paper extends the application of Rényi's $\alpha$ -divergence to sparse high-dimensional linear regression with spike-and-slab priors, a setting not previously explored with this divergence measure.
Algorithmic Derivation:
- Derivation of AlphaVB: A deterministic CAVI algorithm that successfully handles the intractability of the Rényi objective using the Delta method.
- Derivation of AlphaSVB: A stochastic algorithm based on the VR bound and MC approximation.
Hyperparameter Analysis: The study systematically investigates how the choice of $\alpha$ influences variable selection and estimation performance, demonstrating that $\alpha$ is not merely a theoretical parameter but a practical tuning knob for sparsity.

Results
Extensive simulations were conducted across four configurations varying in dimensionality ( $p$ ), sample size ( $n$ ), and sparsity levels ( $s$ ). The proposed methods were benchmarked against state-of-the-art frequentist (Lasso) and Bayesian (SparseVB, VarBVS, SpikeSlab) methods.

Performance of AlphaVB: AlphaVB demonstrated competitive performance across all metrics ( $\ell_2$ error, False Discovery Rate (FDR), True Positive Rate (TPR), and Mean Squared Prediction Error (MSPE)). It achieved particularly low FDR (around 0.01–0.02) and high TPR, indicating effective sparsity modeling. The authors note that while AlphaVB's $\ell_2$ error was slightly higher than the best-performing method in specific settings, it remained competitive without anomalies.
Performance of AlphaSVB: In contrast, AlphaSVB performed significantly worse than AlphaVB and other benchmarks across all configurations. The authors attribute this to the difficulty of direct Monte Carlo optimization in sparse, high-dimensional settings coupled with their specific model specification.
Sensitivity to $\alpha$ :
- For AlphaVB, small values of $\alpha$ close to 1 (specifically $\alpha = 1.01$ ) yielded the best overall balance of metrics. Larger $\alpha$ values (e.g., $\alpha \ge 5$ ) drastically improved FDR (approaching 0) but severely degraded TPR and estimation accuracy, illustrating a trade-off between strict sparsity and signal recovery.
- For AlphaSVB, performance was generally poor regardless of $\alpha$ , though values $\alpha < 1$ (e.g., $\alpha = 0.9$ ) showed slightly more consistent results than $\alpha > 1$ .

Significance and Claims
The paper positions itself as a pilot study for the broader adoption of $\alpha$ -divergence in Variational Inference for high-dimensional problems. The authors claim that:

Flexibility: The introduction of the $\alpha$ hyperparameter provides a mechanism to tune the trade-off between zero-forcing and mass-covering behaviors, which is crucial for different sparsity configurations.
Viability of AlphaVB: The Delta method approximation successfully enables the use of Rényi divergence in a coordinate ascent framework, producing a method (AlphaVB) that is computationally efficient (seconds to minutes) and statistically competitive with existing state-of-the-art procedures.
Practical Guidance: The study provides empirical evidence that $\alpha \approx 1.01$ is a robust default for AlphaVB in sparse regression, while acknowledging that no single $\alpha$ value is optimal for all metrics or configurations.

The authors conclude that while AlphaVB is a compelling choice for future applications, the limitations observed in AlphaSVB highlight the challenges of MC approximation in this specific context. They suggest future work should focus on extending these methods to Generalized Linear Models (GLMs) and improving computational efficiency through hybrid implementations (e.g., C++/Rcpp).

More like this