📊 statistics

SEMMS with Random Effects: A Mixed-Model Extension for Variable Selection in Clustered and Longitudinal Data

This paper extends the SEMMS variable-selection procedure to handle clustered and longitudinal data by incorporating random effects through an alternating coordinate-ascent algorithm, demonstrating significantly improved accuracy in identifying true predictors compared to the original method when random-effect variance is dominant.

Original authors: Haim Bar, Martin T. Wells

Published 2026-03-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Haim Bar, Martin T. Wells

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Which specific clues (variables) actually caused a crime (the outcome), and which ones are just red herrings?

In the world of data science, this is called Variable Selection. You have a massive list of suspects (predictors), but only a few are actually guilty. The goal is to find the guilty ones without getting distracted by the innocent crowd.

This paper introduces an upgrade to a detective tool called SEMMS. Here is the story of how it works, why it needed an upgrade, and how the new version solves a tricky problem.

1. The Original Detective (Plain SEMMS)

The original SEMMS tool was brilliant at finding the "guilty" clues in a list of suspects. It used a clever statistical trick (a "mixture prior") to say, "Most of these clues are noise, but a few are real signals. Let's find the real ones."

The Flaw: The original tool assumed that every piece of evidence was independent. It thought, "This clue from Suspect A has nothing to do with this clue from Suspect B."

The Real World Problem: In reality, data often comes in clusters.

Imagine you are measuring the reaction times of 18 people over 10 days.
Person A might naturally be slow. Person B might be naturally fast.
If you measure Person A on Day 1 and Day 2, those two measurements are correlated because they come from the same person. They share a "personality" (a random effect).

If you ignore this, your detective tool gets confused. It sees the pattern of "Person A is always slow" and thinks, "Aha! This is a strong signal!" But it's not a signal about the crime; it's just the background noise of that specific person. The original tool would get distracted by these "personality traits" and pick the wrong suspects.

2. The New Detective (Mixed-Model SEMMS)

The authors, Haim Bar and Martin Wells, realized they needed to teach the detective how to handle groups. They created Mixed-Model SEMMS.

Think of the new tool as a detective with a two-step cleaning process:

Step 1: The "Group Filter" (Removing the Noise)

Before looking for the clues, the new tool first identifies the "group personality."

It asks: "How much of this data is just because it came from Person A? How much is just because it came from School B?"
It calculates a "baseline" for each group (like a random intercept or slope).
It then subtracts this baseline from the data.

The Analogy: Imagine you are trying to hear a whisper in a noisy room.

Old Way: You try to listen to the whisper while the whole room is shouting. You might mistake the shouting for the whisper.
New Way: You put on noise-canceling headphones that specifically cancel out the shouting of the specific person sitting next to you. Now, the room is quiet, and you can hear the whisper clearly.

Step 2: The "Clue Hunt" (Variable Selection)

Once the "group noise" is filtered out, the tool runs its original, brilliant algorithm on the cleaned data.

Now, when it sees a pattern, it knows it's a real signal, not just a group quirk.
It picks the true suspects with much higher accuracy.

3. How It Works in Practice (The "Alternating Dance")

The paper describes a clever "dance" the algorithm does to get this right:

Guess the Group Noise: It estimates the group baselines (e.g., how slow Person A is).
Clean the Data: It subtracts that noise.
Find the Clues: It runs the selection algorithm on the clean data to find the important variables.
Refine the Noise: Now that it knows the important variables, it re-estimates the group baselines more accurately.
Repeat: It keeps dancing back and forth until the answer stabilizes.

4. Why This Matters (The Results)

The authors tested this new tool in three different "crime scenes" (simulations):

Scenario A (Strong Signal, Moderate Noise): When the clues were loud and clear, the new tool worked just as well as the old one. No big loss.
Scenario B (Weak Signal, Loud Noise): This is where the old tool failed miserably. The "group noise" was so loud it drowned out the clues. The old tool picked the wrong suspects 99% of the time. The new tool, however, filtered the noise and found the right suspects 93% of the time.
Scenario C (Tiny Data, Huge List): When there were very few people but thousands of clues, the new tool still managed to separate the group noise from the real signals, while the old tool got confused.

They also tested it on non-numerical data (like "Yes/No" answers or counts of things), showing the tool is versatile enough to handle different types of mysteries.

5. The Real-World Test: The Sleep Study

To prove it works in real life, they used a famous dataset about sleep deprivation.

The Setup: 18 people stayed awake for days. Their reaction times were measured.
The Trick: They added 50 fake "clues" (random numbers) to the data. Only 2 of them were actually linked to reaction time.
The Result:
- The Old Tool picked the 2 real clues but also picked 1 fake one (a false alarm).
- The Plain Lasso (a popular competitor) picked the 2 real clues but picked 17 fake ones.
- The New Mixed-Model Tool picked exactly the 2 real clues and zero fake ones.

The Bottom Line

This paper is about teaching a smart data tool to ignore the background chatter of groups so it can focus on the real story.

If you have data where people are measured multiple times, or students are in the same school, or patients are in the same hospital, the old tools get confused by the "group personality." The new Mixed-Model SEMMS puts on noise-canceling headphones, filters out the group effects, and finds the true signals with incredible precision. It turns a messy, confusing pile of data into a clear, solvable mystery.

1. Problem Statement

The paper addresses a critical limitation in the SEMMS (Scalable Empirical-Bayes Model for Marker Selection) algorithm. While the original SEMMS is a powerful variable-selection procedure for generalized linear models (GLMs) using a three-component normal mixture prior, it assumes that all observations are independent.

In real-world applications such as longitudinal studies, repeated-measures experiments, and clustered data (e.g., patients within hospitals, students within schools), observations within the same subject or cluster are correlated. Ignoring this correlation leads to:

Inflated apparent residual variance: The model mistakes within-subject correlation for noise.
Degraded variable selection: True predictors become harder to detect, and noise predictors are more likely to be selected as false positives.
Failure in high-variance regimes: When random-effect variance dominates the fixed-effect signal, standard SEMMS fails to recover the true predictor set.

2. Methodology

The authors propose an extension of SEMMS to accommodate random intercepts and random slopes via a coordinate-ascent algorithm. The core innovation is the decoupling of fixed-effect variable selection from random-effect estimation.

The Extended Model

The model is defined as:
$Y_{ij} = x_{ij}^\top \beta + z_{ij}^\top \alpha + b_{0i} + b_{1i} t_j + \epsilon_{ij}$
Where:

$\beta$ : Fixed effects for covariates to be retained.
$\alpha$ : Coefficients for candidate predictors (selected via the mixture prior).
$b_{0i}, b_{1i}$ : Subject-specific random intercept and slope.
$\epsilon_{ij}$ : Independent observation noise.

The Alternating Coordinate-Ascent Algorithm

The algorithm iterates between two steps until convergence:

Fixed-Effect Step (Variable Selection):
- Compute the Best Linear Unbiased Predictors (BLUPs) of the random effects ( $\hat{u}$ ) from the previous iteration.
- Construct an adjusted response: $Y^* = Y - \hat{u}$ .
- Apply the standard GAM (Generalized Alternating Maximization) algorithm to $Y^*$ . This step treats $Y^*$ as independent, allowing the original SEMMS logic (Woodbury identity for $O(NL^2)$ complexity) to run unchanged.
Random-Effect Step (Parameter Update):
- Fit a linear mixed model (lmer for Gaussian) or generalized linear mixed model (glmer for non-Gaussian) using the currently selected active set of predictors.
- Update the random-effect parameters and the BLUPs ( $\hat{u}$ ).

Non-Gaussian Extension (Poisson & Binomial)

For non-Gaussian families, the authors adapt the Penalized Quasi-Likelihood (PQL) approach (Breslow & Clayton, 1993):

Instead of subtracting raw random effects, they construct a link-scale adjusted working response ( $W^*$ ).
This involves a first-order Taylor expansion of the canonical link function around the current fitted values.
The fixed-effect step then runs a standard Gaussian SEMMS on this continuous working response $W^*$ .

3. Key Contributions

Algorithmic Extension: A novel alternating algorithm that integrates random effects into the SEMMS framework without modifying the core $O(NL^2)$ computational engine.
Warm-Start Strategy: The algorithm initializes the random-effect offset using a null mixed model fit. This prevents the initial variable selection step from absorbing subject-level variation into false positive predictors.
Non-Gaussian Adaptation: A rigorous derivation of the working-response adaptation for Poisson and Binomial data within the alternating loop, ensuring the method applies to count and binary outcomes.
Implementation: The extension is implemented in the R package SEMMS (v0.4.0), maintaining backward compatibility and leveraging the lme4 package for mixed-model fitting.

4. Simulation Results

The authors conducted extensive simulations across Gaussian, Poisson, and Binomial distributions, varying signal strength, random-effect magnitude, and sample dimensions ( $N$ vs. $K$ ).

Key Findings:

Dominant Random Effects (Weak Signal): This is the most critical scenario.
- Gaussian: Mixed SEMMS achieved 93% exact recovery of the true predictor set, compared to only 1% for plain SEMMS.
- Poisson: Mixed SEMMS achieved 61% exact recovery vs. 45% for plain SEMMS.
- Binomial (Dominant RE): Mixed SEMMS achieved 65% exact recovery vs. 39% for plain SEMMS.
Strong Signal: When fixed effects are strong relative to random effects, both methods perform comparably, though Mixed SEMMS maintains slightly lower false-positive rates.
High-Dimensional ( $K > N$ ): In scenarios where predictors exceed observations (e.g., $K=200, N=90$ ), Mixed SEMMS significantly outperformed plain SEMMS and Lasso-based methods, achieving 91% exact recovery vs. 80% for plain SEMMS.
Comparison with Competitors:
- Plain Lasso: Consistently suffered from high false-positive rates because it ignored correlation structures.
- glmmLasso: Often failed due to over-penalization by BIC when random-effect variance was large, leading to under-selection (missing true predictors).

Real-Data Example:
Using an augmented version of the sleepstudy dataset, Mixed SEMMS was the only method to achieve exact recovery of the two true synthetic predictors with zero false positives. Plain SEMMS included one false positive, and glmmLasso selected no variables.

5. Significance

This work bridges a critical gap between scalable empirical Bayes variable selection and mixed-effects modeling.

Robustness: It provides a robust solution for variable selection in clustered data where ignoring correlation leads to severe statistical errors.
Efficiency: By utilizing the Woodbury identity and coordinate ascent, it retains the computational scalability of the original SEMMS, making it feasible for high-dimensional genomic and longitudinal datasets.
Practical Utility: The method is particularly vital for fields like genomics, pharmacology, and psychology, where subject-specific heterogeneity is the norm, and identifying true biomarkers or predictors requires disentangling them from random noise.

In summary, the Mixed-Model SEMMS extension transforms a powerful variable-selection tool into a state-of-the-art solution for clustered and longitudinal data, significantly outperforming existing methods when random-effect variance is substantial.