📊 statistics

Variational Bayesian Sparse Negative Binomial Regression

This paper introduces a computationally efficient variational Bayesian framework for sparse negative binomial regression that achieves MCMC-level accuracy with less than 1% of the computation time, offering robust performance for high-dimensional overdispersed count data while outperforming Poisson-based approaches in such settings.

Original authors: Mitra Kharabati, Morteza Amini, Mohammad Arashi

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mitra Kharabati, Morteza Amini, Mohammad Arashi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of fingerprints or footprints, your clues are numbers. Specifically, you are dealing with count data: things you can only count on your fingers, like the number of times a bird visits a feeder, the number of typos in a novel, or the number of times a teenager checks their phone. In the world of statistics, there's a classic tool called Poisson regression used to predict these counts. It works on a simple rule: if the average number of events is 5, the "wiggle room" or variation around that average is also 5.

But real life is messy. Sometimes, the numbers are wilder than that. Maybe the bird feeder gets visited 5 times on average, but one day it's 0 and the next it's 50. This is called overdispersion, where the chaos (variance) is much bigger than the average. To handle this, statisticians use a more flexible tool called Negative Binomial regression. However, when you have thousands of clues (predictors) and only a few suspects (data points), the math gets so heavy that the super-computers used to solve it (called MCMC) take forever to run. This is where Variational Bayes comes in: it's like a smart shortcut that guesses the answer quickly by turning a hard math problem into a simpler optimization game, trading a tiny bit of perfect precision for massive speed.

This paper, titled "Variational Bayesian Sparse Negative Binomial Regression," is about building a super-fast, super-smart detective kit for these messy, high-stakes counting mysteries. The authors, Mitra Kharabati, Morteza Amini, and Mohammad Arashi, realized that while existing fast methods were great for simple cases, they often failed when the data was "overdispersed" (wildly variable). They decided to build a new framework that combines the flexibility of the Negative Binomial model with a "sparse" approach—a way to automatically ignore the useless clues and focus only on the ones that actually matter.

The team developed two new methods, one using a "Horseshoe" prior and another using a "Continuous Shrinkage" prior. Think of these as magical filters. The Horseshoe filter is like a sieve that lets the big, important signals pass through while crushing the tiny, noisy ones into dust. The Continuous Shrinkage filter does something similar but with a slightly different mechanism, gently squeezing the unimportant numbers down to zero. The goal was to see if these fast "Variational Bayes" (VB) shortcuts could do the same job as the slow, heavy "MCMC" super-computers, but in a fraction of the time.

Here is what they found. In their simulations, which involved creating thousands of fake datasets to test their tools, the new VB methods were incredibly accurate. They managed to estimate the true numbers and pick out the right variables just as well as the slow MCMC methods. But the real kicker? They did it in less than 1% of the time. If the MCMC method took 100 hours to solve a puzzle, the new VB method solved it in less than an hour.

Crucially, the paper rules out a common shortcut: using the simpler Poisson model when the data is actually overdispersed. The authors showed that if you try to use the Poisson model on wild, overdispersed data, your results fall apart. The error rates skyrocket, and your predictions become unreliable. It's like trying to measure a hurricane with a ruler meant for a gentle breeze. The paper demonstrates that their Negative Binomial approach is essential for this kind of data. Interestingly, they also found that their new method is robust; even if the data was perfectly calm (Poisson), their method still worked well, making it a safer "default" choice for scientists who aren't sure what kind of data they are dealing with.

The researchers didn't just stop at fake data. They tested their methods on real-world datasets, including records of extramarital affairs, bike-sharing rentals, and hospital stay lengths. In every case, the Negative Binomial models fit the data much better than the Poisson models, confirming that real-world count data is often wild and overdispersed. The new VB methods provided predictions that were just as good as the heavy-duty MCMC benchmarks but were fast enough to be practical for everyday use.

In the end, this paper suggests that we don't have to choose between speed and accuracy anymore. By using these new Variational Bayesian tools, researchers can now handle complex, high-dimensional counting data quickly and reliably, without needing to wait days for a computer to finish its calculations. It's a win for anyone trying to make sense of the chaotic, countable world around them.

Technical Summary: Variational Bayesian Sparse Negative Binomial Regression

Problem Statement
Modern applications in fields such as epidemiology, ecology, and finance frequently involve count data characterized by overdispersion (where variance exceeds the mean) and high-dimensional predictors (where the number of variables $p$ exceeds the sample size $n$ ). While the Negative Binomial (NB) regression model is the standard flexible framework for handling overdispersion, existing Bayesian approaches for sparse NB regression rely heavily on Markov chain Monte Carlo (MCMC) methods. These MCMC methods are computationally prohibitive in high-dimensional settings, often rendering them impractical for large-scale data analysis. Furthermore, while Variational Bayes (VB) has been successfully applied to sparse Poisson regression, a comprehensive VB framework for sparse NB regression that simultaneously addresses overdispersion and variable selection has been lacking.

Methodology
The paper proposes a Variational Bayesian (VB) framework for sparse negative binomial regression using a mean-field approximation. The core methodological innovations include:

Model Formulation: The NB likelihood is re-parameterized using an auxiliary variable approach, where $y_i | g_i \sim \text{Poisson}(g_i)$ and $g_i | \beta, \kappa \sim \text{Gamma}(\kappa, \kappa \exp(-X_i\beta))$ . This facilitates the derivation of variational updates.
Sparsity-Inducing Priors: Two specific priors are implemented to enforce sparsity in the regression coefficients $\beta$ $β$ :
- Continuous Spike-and-Slab (CS) Prior: A continuous relaxation of the discrete spike-and-slab prior, utilizing a mixture of two Gaussians with different variances.
- Horseshoe (HS) Prior: A global-local shrinkage prior that places Half-Cauchy distributions on local and global scale parameters, allowing for strong shrinkage of noise while preserving large signals.
Non-Conjugate Variational Inference: Due to the non-conjugate relationship between the dispersion parameter and regression coefficients in the NB model, the authors employ the non-conjugate mean-field VB framework (Wand et al., 2011). This involves assuming exponential family distributions for parameter blocks and deriving fixed-point update equations for the natural parameters (means and covariances) to maximize the Evidence Lower Bound (ELBO).
Sparsity Enforcement: Since VB posterior means are not inherently sparse, the authors apply a hard-thresholding strategy based on the Akaike Information Criterion (AIC) to select variables.
Posterior Predictive Approximation: To compute the posterior predictive mass function, the authors utilize a second-order Taylor expansion of the integrand around the variational mean of the dispersion parameter, approximating the integral using properties of the log-normal distribution.

Key Contributions

Algorithmic Development: The paper derives closed-form variational update equations for both CS and HS priors within the NB regression framework, addressing the computational challenges of non-conjugacy.
Computational Efficiency: The proposed VB algorithms achieve estimation accuracy and variable selection performance comparable to MCMC benchmarks while requiring less than 1% of the computation time.
Robustness Analysis: The study demonstrates that while Poisson-based VB methods fail under overdispersion, the proposed NB methods remain robust even when data are generated from a Poisson distribution, suggesting they are a safer default choice.
Scalability: The framework is shown to be effective in high-dimensional scenarios ( $p \gg n$ ) where MCMC methods become intractable.

Results
Extensive simulation studies and real-data analyses were conducted to evaluate the proposed methods (HS-VB-NB and CS-VB-NB) against frequentist penalized methods (LASSO, SCAD), Poisson-based VB methods, and MCMC benchmarks.

Simulation (Low-Dimensional): In settings with $n=100, p=10$ , the NB-VB methods achieved coefficient relative errors (CRE) and test relative errors (TSRE) comparable to MCMC. In contrast, Poisson-based VB methods exhibited significantly higher errors due to model misspecification. The NB-VB methods also provided coverage probabilities for Highest Posterior Density (HPD) intervals close to the nominal 0.95 level, whereas Poisson-VB coverage was severely deficient (0.14–0.29).
Simulation (High-Dimensional): In settings with $n=30, p=200$ , the NB-VB methods outperformed Poisson-VB methods across all metrics (CRE, FPR, FNR, CRPS). The NB-VB methods maintained low false positive rates and accurate variable selection, while MCMC was excluded from these comparisons due to prohibitive computational costs.
Real Data Applications: The methods were applied to five benchmark datasets (Affairs, Bike Sharing, and three Arizona hospital datasets) and a Fishing dataset.
- Overdispersion tests confirmed the necessity of the NB model for all datasets.
- The NB-VB methods demonstrated competitive predictive performance (measured by Test Relative Error and CRPS) against frequentist LASSO and SCAD methods.
- On the Fishing dataset, the VB methods closely replicated the predictive performance of MCMC benchmarks with substantial computational savings.
- The CS-VB-NB method generally exhibited the lowest median test errors and high stability across datasets.

Significance and Claims
The paper claims to fill a critical gap in the literature by providing the first comprehensive variational Bayesian framework for sparse negative binomial regression. The authors assert that their approach offers a unified solution for overdispersion, variable selection, and parameter estimation in high-dimensional settings.

The significance of the work lies in its ability to provide computationally efficient inference without sacrificing the accuracy and uncertainty quantification typically associated with MCMC. The authors emphasize that the negative binomial specification is essential for overdispersed data, as Poisson-based approaches suffer substantial performance degradation in such contexts. Conversely, the proposed NB methods are claimed to be robust even when data is Poisson-distributed, making them a reliable "safer default" for practitioners. The paper concludes that the framework provides a practical tool for sparse count regression, enabling the analysis of large-scale datasets that were previously computationally inaccessible to Bayesian sparse methods.

Limitations Acknowledged
The authors modestly note that the mean-field assumption may underestimate posterior uncertainty for parameters with strong dependencies. Additionally, the quadratic approximation used for the non-conjugate NB model may not capture highly non-Gaussian posterior shapes. The performance is also noted to depend on the choice of the sparsity-inducing prior, with the Horseshoe prior showing slightly better estimation accuracy and the Continuous Shrinkage prior showing more stable convergence.

Technical Summary: Variational Bayesian Sparse Negative Binomial Regression

More like this