📊 statistics

A Goodness-of-Fit Test for Mixed-Effects Logistic Regression

This paper introduces a data-driven grouping-based Wald test implemented in Stata that extends goodness-of-fit assessment to mixed-effects logistic regression models with random slopes, effectively maintaining nominal Type I error and detecting fixed-effects misspecification even in sparse cluster settings where previous methods fail.

Original authors: Ariel Linden

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ariel Linden

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor trying to predict whether a patient will recover from an illness. You have data on thousands of patients, but these patients aren't just floating in a vacuum; they are grouped. Some patients belong to the same family, others are treated by the same doctor, and some are in the same hospital.

In statistics, this is called Mixed-Effects Logistic Regression. It's a fancy way of saying, "Let's predict a Yes/No outcome (like recovery) while acknowledging that people in the same group tend to be more similar to each other than to people in other groups."

For a long time, statisticians had a problem: How do you know if your prediction model is actually any good?

The Problem: The "One-Size-Fits-All" Ruler

For simple data (where everyone is independent), statisticians have a tool called the Hosmer-Lemeshow test. Think of this like a ruler. To check if your model fits, the ruler divides all your patients into 10 buckets based on how likely they are to recover. Then, it checks: "Did the model predict the right number of recoveries in each bucket?"

However, this ruler had two major flaws when applied to complex, grouped data:

It couldn't handle "Slopes": Sometimes, a treatment works differently for different families. Maybe it works great for Family A but poorly for Family B. Old tests couldn't check models that allowed for these differences.
It broke on small groups: The ruler always tried to make 10 buckets. But what if you only have 5 patients in a specific family? You can't put 5 people into 10 buckets without leaving some empty. The old method would crash and give up, leaving researchers with no way to check their work.

The Solution: A New, Flexible Ruler

Ariel Linden, the author of this paper, invented a new tool called mlm_gof. Think of this as a smart, adjustable ruler that fixes both problems.

1. The "Smart Bucket" Rule (The Data-Driven Rule)

Instead of forcing 10 buckets no matter what, this new tool looks at your smallest group first.

The Analogy: Imagine you are sorting marbles into jars. If your smallest jar only holds 3 marbles, you don't try to make 10 jars; you just make 3 jars.
The Rule: The tool says, "I will make as many buckets as the smallest group has people, but I'll never make more than 10."
- If your smallest group has 20 people, it makes 10 buckets (the standard).
- If your smallest group has only 5 people, it makes 5 buckets.
Why it matters: This prevents the "crash" that happened with the old method. It ensures the test works even when you have small, sparse groups of data.

2. Checking the "Slopes" (Random Effects)

The new tool can now handle models where the effect of a treatment changes from group to group.

The Analogy: Imagine you are testing a new diet. The old test assumed the diet worked the exact same way for everyone. The new test acknowledges that for some families, the diet might be a magic bullet, while for others, it might do nothing. It checks if the model correctly captures these different "personalities" of the groups.

How the Test Works (The "Stress Test")

Here is how the new tool checks if your model is good, step-by-step:

Make Predictions: The model guesses the probability of recovery for every single person.
Sort and Group: It sorts people within their specific families into "buckets" based on how likely they are to recover.
Add a "Trap": It adds a special "trap" variable to the math. This variable asks: "Hey, are there any patterns in the buckets that the model missed?"
The Verdict: It runs a statistical check (a Wald test).
- If the result is "No" (High p-value): The trap didn't catch anything. The model is doing a great job; the predictions match reality.
- If the result is "Yes" (Low p-value): The trap caught something! The model is missing something important (like a non-linear relationship or a missing interaction).

What the Simulations Showed

The author ran thousands of computer experiments (simulations) to see if this new ruler works.

It doesn't cry wolf: When the model was perfect, the test rarely said it was bad (it kept the "Type I error" low).
It catches mistakes: When the model was missing a key piece of the puzzle (like a curved relationship or a missing interaction), the test got very good at spotting it.
It has a blind spot: The test is great at finding errors in the predictors (the variables you put in), but it cannot tell you if you missed an entire level of grouping.
- Analogy: If you forgot to account for the fact that patients are grouped by hospital, this test won't necessarily tell you that. It's like checking if the recipe is right, but not noticing you forgot to mention you were cooking for a whole army instead of a family. For that, you need a different tool.

The Takeaway

This paper gives researchers a reliable, flexible checklist for their complex medical and social science models.

Before: If you had small groups or complex models, you were flying blind, hoping your model was right.
Now: You have a tool (mlm_gof) that automatically adjusts to your data size and checks if your model is actually describing reality.

It's like upgrading from a rigid, breakable ruler to a smart, flexible measuring tape that ensures your scientific conclusions are built on solid ground.

1. Problem Statement

Mixed-effects logistic regression is the standard tool for analyzing binary outcomes in hierarchical data (e.g., patients within doctors, repeated measures within individuals). However, assessing the goodness-of-fit (GOF) for these models remains a significant challenge:

Limited Scope: Existing formal GOF tests (e.g., extensions of the Hosmer–Lemeshow test) are restricted to random intercept models. They fail to accommodate random slopes, where the effect of a covariate varies across clusters.
Sparse Cluster Issues: Traditional methods rely on a fixed rule of dividing observations into 10 groups ( $G=10$ ). In mixed-effects settings, clusters often contain fewer than 10 observations. This leads to empty cells, degenerate indicator variables, and estimation failure.
Gap in Literature: There is no robust, formal GOF test that handles both random slopes and sparse cluster sizes simultaneously.

2. Methodology

The author proposes a new GOF test based on a grouping-based Wald statistic, adapted for mixed-effects models with random slopes.

The Test Procedure

The method follows a six-step process implemented in the Stata program mlm_gof:

Baseline Model: Fit the mixed-effects logistic model (including random intercepts and slopes) using maximum likelihood estimation (MLE) to obtain predicted probabilities ( $\hat{p}_{ij}$ ).
Data-Driven Group Selection: Instead of fixing $G=10$ , the number of groups is determined by the rule:
$G = \min(10, n_{min})$
where $n_{min}$ is the size of the smallest cluster. This ensures every group contains at least one observation per cluster, preventing estimation failure in sparse settings.
Grouping: Within each cluster, observations are sorted by predicted probability and assigned to $G$ approximately equal-sized groups.
Augmented Model: Create $G-1$ binary indicator variables representing these groups. These indicators are added to the original model as fixed effects.
Refitting: Estimate the augmented model retaining the original random-effects structure.
Wald Statistic: Test the joint significance of the group indicators ( $H_0: \gamma_2 = \dots = \gamma_G = 0$ ) using a Wald statistic ( $W$ ), which follows a $\chi^2$ distribution with $G-1$ degrees of freedom.

Simulation Design

The method was validated via an extensive simulation study (44,000 replications) covering:

Three-level models: Family $\to$ Subject $\to$ Repeated measures.
Random Slopes: Included in the true data-generating process.
Scenarios: Varied cluster sizes ( $J=15, 30, 50$ ), cluster counts, and Intraclass Correlation Coefficients (ICC).
Misspecification Types: Omitted quadratic terms, omitted interactions, and omitted clustering levels.

3. Key Contributions

Extension to Random Slopes: The test is the first formal GOF procedure capable of evaluating mixed-effects logistic models where regression coefficients (slopes) vary randomly across clusters.
Data-Driven Grouping Rule: The introduction of $G = \min(10, n_{min})$ solves the critical issue of test failure in sparse clusters. The paper demonstrates that the conventional $G=10$ rule leads to complete test failure in balanced designs with small clusters ( $n < 10$ ).
Implementation: The method is operationalized in the community-contributed Stata program mlm_gof, making it accessible for applied researchers.

4. Key Results

The simulation study yielded three primary findings:

Type I Error Control: The test maintains the nominal Type I error rate (0.05) across 24 null scenarios, including the smallest sample sizes tested ( $N=1,500$ ). This performance is stable even where previous studies reported instability.
Power to Detect Misspecification:
- Omitted Nonlinearity: The test showed increasing power to detect omitted quadratic terms, rising from 0.07 (negligible effect) to 1.00 (strong effect).
- Omitted Interactions: Power reached 0.87 for strong interaction effects.
- Omitted Clustering Level: As expected, the test had no power to detect the omission of a clustering level (e.g., fitting a 2-level model when the data is 3-level). This is a structural limitation, as the test detects patterns in conditional predicted probabilities, which are not necessarily altered by missing random intercepts.
Group Selection Sensitivity:
- In unbalanced designs, both the data-driven rule and fixed $G=10$ performed similarly (dominated by larger clusters).
- In balanced designs with small clusters ( $n < 10$ ), fixing $G=10$ resulted in 100% test failure, whereas the data-driven rule produced valid results in all replications.

5. Significance and Implications

Practical Utility: The mlm_gof command fills a critical gap in the diagnostic toolkit for multilevel binary data analysis, allowing researchers to formally assess model fit in complex random-slope scenarios.
Methodological Advancement: By utilizing full maximum likelihood estimation (via adaptive Gauss–Hermite quadrature) rather than penalized quasi-likelihood (PQL/MQL), the test provides more reliable parameter estimates, particularly in small samples.
Guidance for Researchers: The paper clarifies that while this test is excellent for detecting fixed-effects misspecification (nonlinearity, interactions), it should be paired with likelihood-based model comparisons (e.g., AIC/BIC) to assess the adequacy of the random-effects structure itself.
Robustness: The proposed data-driven grouping rule ensures that GOF testing is feasible even in studies with small clusters, a common scenario in health services and social science research.

In conclusion, this work provides a principled, robust, and implementable solution for validating mixed-effects logistic regression models, extending the applicability of GOF testing to the more general and complex case of random slopes and sparse data.