📊 statistics

Fitting Large Nonlinear Mixed Effects Models Using Variational Expectation Maximization

This paper introduces and validates the Variational Expectation Maximization (VEM) algorithm as a scalable alternative for fitting large Nonlinear Mixed Effects (NLME) models, demonstrating its ability to efficiently handle complex datasets with over 15,000 population parameters using the Pumas software.

Original authors: Mohamed Tarek, Pedro Afonso

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohamed Tarek, Pedro Afonso

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a massive mystery involving thousands of suspects. Each suspect has their own unique quirks (random effects), but they all follow a general rulebook written by the police chief (population parameters). Your goal is to figure out exactly what that rulebook says, even though you can only see the suspects' actions, not their internal thoughts.

In the world of medicine and drug testing, this is exactly what scientists do with Nonlinear Mixed Effects (NLME) models. They try to understand how a drug works in a whole population while accounting for how it works differently in every single person.

The problem? As the number of suspects (people) and the complexity of the rulebook (parameters) grow, the math becomes so heavy that traditional computers get stuck. It's like trying to solve a puzzle where every piece is a giant boulder; you can't move them fast enough to finish the picture.

This paper introduces a new, lighter way to solve this puzzle called Variational Expectation Maximization (VEM). Here is how it works, using simple analogies:

1. The Old Way: The "Guess and Check" Hike

Traditional methods (like FOCE) are like trying to find the bottom of a foggy valley by walking every single path. You have to stop, look around, and calculate the slope at every single step to make sure you aren't going the wrong way. If the valley is huge (a model with 15,000 parameters), this hike takes forever. You might spend 24 hours just trying to take one step, and the computer gives up.

2. The New Way: The "Smart Map" (VEM)

The authors propose VEM, which is like hiring a team of cartographers to draw a simplified map of the valley instead of walking every path.

The Approximation: Instead of calculating the exact, impossible shape of the valley for every person, VEM draws a smooth, simple hill (a "Gaussian" shape) that looks like the real valley. It's not perfect, but it's close enough to get you moving.
The Two-Step Dance:
1. Draw the Map (E-step): The algorithm quickly sketches a best-guess map of where each person's quirks might be.
2. Update the Rulebook (M-step): Using those maps, it updates the main rulebook to see if it fits better.
3. Repeat: It keeps redrawing the maps and updating the rulebook until they match perfectly.

3. The Secret Weapon: "Reverse-Mode" Thinking

The paper highlights a specific trick called Reverse-Mode Automatic Differentiation.

Forward Mode (The Old Way): Imagine you are painting a wall. You paint a little bit, then check the color, then paint a little more. If you have 15,000 colors to mix, you have to check the color 15,000 times for every single brushstroke. It's slow.
Reverse Mode (The New Way): Imagine you are looking at the finished wall and working backward to figure out exactly how much of each color was used. You can calculate the entire recipe for all 15,000 colors in one go. This is why VEM is so fast for huge models.

What Did They Prove?

The authors tested this new method with two experiments:

The Small Test (The Warfarin Model): They used a standard, well-known drug model with 31 people.
- Result: VEM found the same answers as the old method but did it much faster (16 seconds vs. 44 seconds). It proved the new method is accurate.
The Big Test (The DeepNLME Model): They created a "monster" model with 15,410 parameters (like a rulebook with 15,000 pages) and 16 different types of quirks per person.
- Result: The old method (FOCE) tried to take a single step and failed to finish even after 24 hours. The new method (VEM) kept chugging along, completing 5 full steps in a reasonable amount of time.

The Bottom Line

The paper claims that VEM is a powerful new tool that allows scientists to solve massive, complex drug models that were previously too heavy for computers to handle. It doesn't just work for small puzzles; it shines when the puzzle is gigantic.

Important Caveats from the Paper:

It's an approximation: Because it uses a "simplified map," it needs to be tuned carefully (like choosing the right number of samples) to ensure the map is accurate.
It needs a second step: Unlike the old method, VEM doesn't give you the final "score" (log likelihood) automatically; you have to calculate that separately at the end.
It's not magic for small things: For tiny models, the old methods might still be faster because the new method has some "setup time" (compilation) before it starts.

In short, the authors have built a high-speed elevator (VEM) that can carry heavy loads (complex models) up a skyscraper, whereas the old method was a set of stairs that gets stuck when the load gets too heavy.

1. Problem Statement

Nonlinear Mixed Effects (NLME) models are the standard for analyzing hierarchical and longitudinal data in pharmacometrics, Quantitative Systems Pharmacology (QSP), and clinical trials. These models estimate population parameters ( $\theta$ ) by maximizing the marginal likelihood, which requires integrating out subject-specific random effects ( $\eta$ ).

The Core Challenge:

Computational Intractability: As models become more complex (e.g., incorporating neural networks, high-dimensional ODEs, or thousands of parameters), the integration required to compute the marginal likelihood becomes computationally prohibitive.
Limitations of Existing Methods: Traditional algorithms like First-Order Conditional Estimation (FOCE) and Stochastic Approximation EM (SAEM) rely on numerical integration or Markov Chain Monte Carlo (MCMC) sampling. These methods scale poorly with the number of parameters and random effects, often failing to converge or taking excessive time for models with thousands of parameters (e.g., DeepNLME models).

2. Methodology: Variational Expectation Maximization (VEM)

The authors propose using Variational Expectation Maximization (VEM) as a scalable alternative. Instead of sampling from the posterior distribution of random effects (as in MCEM/SAEM), VEM approximates the posterior using a parametric distribution (variational family).

Key Technical Components:

Variational Inference (VI) Framework:
- The goal is to maximize the Evidence Lower Bound (ELBO) rather than the marginal likelihood directly.
- The ELBO is defined as: $\text{ELBO} = \mathbb{E}_{q}[\log p(y, \eta|\theta)] - \text{KL}(q(\eta) || p(\eta|y, \theta))$ .
- Maximizing the ELBO is equivalent to minimizing the Kullback-Leibler (KL) divergence between the variational distribution $q(\eta)$ and the true posterior.
Reparameterization Trick:
- To enable gradient-based optimization, the authors use the reparameterization trick. Random effects are generated as a deterministic transformation of a fixed base distribution (e.g., $\eta = T(\xi; \kappa)$ where $\xi \sim \mathcal{N}(0, I)$ ).
- This allows gradients to flow through the sampling process, enabling the use of efficient optimizers.
Joint Optimization:
- Unlike traditional EM which alternates strictly between E-steps and M-steps, the Pumas implementation performs joint optimization of both the population parameters ( $\theta$ ) and the variational parameters ( $\kappa$ ) simultaneously using gradient-based methods.
Automatic Differentiation (AD):
- Reverse-mode AD is utilized for models with a large number of parameters relative to outputs (common in DeepNLME). This is crucial for efficiently computing gradients of the ELBO with respect to thousands of parameters.
- The implementation supports both forward-mode (for smaller models) and reverse-mode AD.
Optimization Strategies:
- Deterministic VEM: Uses pre-sampled base variables and the L-BFGS optimizer (quasi-Newton) for stable, fast convergence.
- Stochastic VEM: Uses resampling per iteration and the Adam optimizer, suitable for mini-batch processing of large populations.
Handling Constraints: For constrained random effects (e.g., log-normal), the method applies domain transformations (e.g., log-transform) before approximating with a Gaussian variational family.

3. Key Contributions

Scalable Algorithm for NLME: The paper introduces a VEM variant specifically tailored for NLME models, capable of fitting models with >15,000 population parameters and 16+ random effects, a regime where traditional methods fail.
Implementation in Pumas: The algorithm is implemented in the Pumas statistical software, leveraging Julia's high-performance computing capabilities and automatic differentiation.
Theoretical Connections: The authors clarify the relationship between VEM, Variational Autoencoders (VAEs), Probabilistic PCA (PPCA), and the Laplace approximation. They demonstrate that the Laplace method is essentially a second-order Taylor approximation of the VEM objective.
Empirical Validation: The paper provides rigorous benchmarks comparing VEM against FOCE on both small-scale (Warfarin) and large-scale (DeepNLME) models.

4. Experimental Results

The authors conducted experiments on two models using the Pumas software:

A. Warfarin Model (Small Scale Validation)

Setup: Standard PK-PD model with 31 subjects and 464 observations.
Comparison: VEM vs. FOCE.
Results:
- Accuracy: VEM produced parameter estimates and log-likelihoods nearly identical to FOCE (Log Likelihood: -994.5 vs -995.3).
- Speed: VEM was significantly faster (16.6 seconds vs. 44.4 seconds).
- Conclusion: VEM is a correct and efficient alternative for standard pharmacometric models.

B. DeepNLME Friberg Model (Large Scale Scalability)

Setup: An artificially large model incorporating neural networks (MLPs) with 15,410 population parameters and 16 random effects.
Comparison: VEM (5 iterations) vs. FOCE (attempted 1 iteration).
Results:
- VEM: Completed 5 iterations in ~~3092 seconds (~~618 seconds/iteration) using reverse-mode AD.
- FOCE: Failed to complete a single iteration within 24 hours.
- Conclusion: VEM successfully scales to complex, high-dimensional models where FOCE becomes computationally intractable.

5. Significance and Implications

Enabling DeepNLME and QSP: The ability to fit models with thousands of parameters opens the door for DeepNLME (integrating neural networks into mechanistic models) and complex Quantitative Systems Pharmacology (QSP) models, which were previously too computationally expensive to fit as NLMEs.
Shift in Optimization Paradigm: The paper advocates for a shift from MCMC-based sampling (SAEM/MCEM) to gradient-based variational inference for large-scale problems, leveraging modern deep learning infrastructure (reverse-mode AD, GPU/TPU compatibility).
Practical Limitations: The authors acknowledge that VEM is sensitive to hyperparameters (number of samples, variational family choice) and requires separate steps to calculate Empirical Bayes Estimates (EBEs) and log-likelihood approximations post-optimization, unlike FOCE which does this during estimation.
Future Directions: The method is poised to become a standard tool for fitting large joint biomarker models and complex biological systems where traditional methods hit a "wall" of computational cost.

In summary, this paper demonstrates that Variational Expectation Maximization, when combined with reverse-mode automatic differentiation, provides a robust, scalable, and accurate framework for fitting the next generation of complex, high-dimensional nonlinear mixed effects models in pharmacometrics.