📊 statistics

Combined shrinkage of fixed and random effects in linear mixed models using empirical Bayes

This paper proposes a novel Empirical Bayes methodology that automates the joint selection of prior parameters for both fixed and random effects in Linear Mixed Models, improving estimation accuracy and predictive performance through marginal likelihood maximization via Laplace approximation.

Original authors: Matteo Amestoy, R. Vermeulen, Mark A. van de Wiel, Wessel N. van Wieringen

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Matteo Amestoy, R. Vermeulen, Mark A. van de Wiel, Wessel N. van Wieringen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to perfect a secret family recipe for a soup that is being served at a massive international food festival.

To make this soup perfect, you have to deal with two different types of "variables" that can mess up your flavor:

The Fixed Effects (The Recipe): These are the core ingredients—the amount of salt, the type of broth, the heat of the stove. If you change these, the flavor changes for everyone.
The Random Effects (The Kitchens): These are the individual kitchens where the soup is being made. Even with the same recipe, one kitchen might have a slightly different stove, or one chef might have a slightly different way of stirring. These differences are "random" and vary from place to place.

The Problem: The "Too Many Cooks" Dilemma

In modern science (like studying how air pollution affects health), researchers are dealing with massive amounts of data. It’s like trying to run that soup recipe in 500 different kitchens all at once.

Sometimes, you run into a mathematical nightmare:

The "Tiny Sample" Problem: Imagine you only have three kitchens in the whole world. If you try to figure out exactly how much "kitchen style" affects the soup based on only three examples, your math will go haywire. You’ll overreact to tiny differences, thinking a kitchen is "special" when it’s actually just a fluke.
The "Too Many Ingredients" Problem: If you try to track 100 different tiny spices (high-dimensional data) but only have a few bowls of soup to test, you can't tell which spice is actually doing the work.

Standard statistical tools usually "break" here. They either give you wildly inaccurate guesses or simply crash because they can't make sense of the chaos.

The Solution: The "Smart Intuition" Method (Empirical Bayes)

The authors of this paper created a new way to handle this chaos. They call it Empirical Bayes Regularization.

Think of this as giving the chef a "Smart Intuition." Instead of letting the chef guess wildly based on a tiny, messy sample, the method provides a "nudge" toward common sense.

Shrinkage (The Nudge): If the data from one kitchen looks extremely weird, the method says, "Hey, don't jump to conclusions. Let's 'shrink' that estimate back toward the average." It prevents the model from overreacting to outliers.
Joint Selection (The Double Check): Usually, scientists only "nudge" the recipe (fixed effects) or only "nudge" the kitchen differences (random effects). This paper does both at the same time. It looks at the ingredients and the kitchens simultaneously to find the perfect balance.

How they do it: The "Laplace Shortcut"

Calculating the perfect "nudge" is mathematically exhausting—it’s like trying to calculate the exact trajectory of every single bubble in the soup. It would take forever.

The authors used a clever mathematical trick called a Laplace Approximation. Instead of tracking every single bubble, they look at the overall shape of the bubbles and make a very high-speed, very accurate guess. This makes the method fast enough to actually use on real-world, massive datasets.

Why does this matter? (The Real-World Result)

The researchers tested this on a real study about air pollution and blood pressure.

In that study, they had very little data on certain cities. Using old methods, the results were shaky. But when they used their new "Smart Intuition" method:

Better Predictions: They could predict health outcomes much more accurately.
Better Confidence: They weren't just guessing; they knew exactly how much they could trust their own numbers.

In short: This paper provides a mathematical "safety rail" that allows scientists to study incredibly complex, messy, and massive datasets without getting lost in the noise. It allows them to build more sophisticated models that actually work in the real, unpredictable world.

Technical Summary: Combined Shrinkage of Fixed and Random Effects in Linear Mixed Models using Empirical Bayes

Authors: Matteo Amestoy, R. Vermeulen, Mark A. van de Wiel, Wessel N. van Wieringen (April 2026)

1. The Problem: Estimation Instability in Complex LMMs

Linear Mixed Models (LMMs) are essential for analyzing hierarchical or longitudinal data. However, the authors identify a critical bottleneck in modern epidemiological and high-throughput studies: parameter unidentifiability and numerical instability when the model complexity exceeds the available data structure. Specifically, two scenarios impede reliable estimation:

Low-level sparsity: When the number of covariates in the random-effects model exceeds the number of statistical units (e.g., only three cities in a multi-city study).
Low-repetition sparsity: When the number of random-effect covariates exceeds the average number of repeated measurements per unit (e.g., three observations per participant).

While regularization (shrinkage) is widely used for high-dimensional fixed effects, the literature has largely overlooked the necessity of regularizing random-effect variance parameters. Traditional approaches often resort to ad hoc methods, such as splitting data into separate models and using meta-analysis, which can lead to inconsistent variance estimation and suboptimal predictive performance.

2. Methodology: Empirical Bayes via Laplace Approximation

The paper proposes a novel, fully data-driven Empirical Bayes (EB) framework for the simultaneous selection of prior parameters for both fixed and random effects.

Model Specification: The authors utilize a Bayesian LMM formulation with conditional conjugate priors:
- Fixed Effects ( $\beta$ ): A Gaussian prior $\mathcal{N}(0, \sigma^2_\lambda I_p)$ , where $\lambda$ acts as a ridge penalty.
- Random Effects Covariance ( $\Sigma_r$ ): An Inverse-Wishart prior $W^{-1}(\eta_r, \Phi_r)$ .
- Residual Variance ( $\sigma^2$ ): A non-informative Inverse-Gamma prior.
Hyperparameter Optimization: Instead of manual selection, the hyperparameters $\Theta$ are chosen by maximizing the marginal likelihood $ML(\Theta)$ . To ensure computational efficiency in high-dimensional settings, the authors use a Laplace approximation of the marginal likelihood rather than computationally expensive MCMC or INLA methods.
MAP Estimation: The Maximum A Posteriori (MAP) estimator is computed using a variant of the Expectation-Maximization (EM) algorithm. This two-step process (E-step for conditional expectations and M-step for parameter updates) allows for efficient convergence even when the parameter space is large.
Intuitive Reparameterization: The authors introduce a reparameterization of the Inverse-Wishart prior using a "prior strength" parameter $b_r$ . This allows practitioners to intuitively understand the balance between the data-driven MLE and the prior mode.

3. Key Contributions

Joint Regularization: A unified framework that shrinks both fixed-effect coefficients and random-effect covariance matrices simultaneously.
Computational Efficiency: The use of Laplace approximation combined with an EM-style algorithm makes the method scalable to high-dimensional data where standard Bayesian methods fail.
Automated Prior Selection: Eliminates the need for subjective or "expert" prior specification for complex random-effect structures.

4. Results

The methodology was validated through extensive simulations and a real-world application:

Simulation Study:
- Variance Reduction: The EB estimator exhibited significantly lower variance in random-effect covariance estimates compared to the Maximum Likelihood Estimator (MLE), particularly for off-diagonal terms.
- Inference Precision: While fixed-effect point estimates remained robust, the EB method provided much more accurate estimates of the fixed-effect variance. This is crucial for valid $p$ -values and significance testing.
- Predictive Accuracy: The EB approach consistently outperformed MLE in terms of Kullback-Leibler (KL) divergence, meaning it produced better probability distributions for new observations.
Real-World Application (Air Pollution & Health):
- Applied to a dataset studying the impact of soot on blood pressure across three cities.
- The EB method allowed for the estimation of a "complex" model (including temperature as a random slope and city-level effects) that outperformed the standard "meta-analysis" approach used in previous literature.
- The regularized complex model achieved predictive performance (RMSE) comparable to simpler, non-regularized models, proving that complexity can be managed without losing accuracy.

5. Significance

This paper provides a bridge between traditional frequentist LMMs and complex Bayesian modeling. By automating the regularization of random effects, it enables researchers to use more sophisticated, hierarchical models in "data-poor" scenarios (small number of clusters or few repetitions) without the risk of numerical instability. This has profound implications for epidemiology and environmental science, where hierarchical structures are common but data at higher levels (like cities or hospitals) are often sparse.