Deep Generalised Mixed Models: a Novel Neural Network Structure for Analysing Hierarchical Data

Technical Summary: Deep Generalised Mixed Models (DGMM)

Problem Statement
The paper addresses the challenges of analyzing intensive longitudinal data, specifically Experience Sampling Method (ESM) data, which involves frequent repeated measurements of thoughts, emotions, and behaviors. Current analytical approaches face a dichotomy:

1. Standard Statistical Techniques: While Mixed Effects Models (MEMs) handle unbalanced data and drop-out well, they often struggle with the high-dimensional, complex correlation structures inherent in ESM data and rely on rigid distributional assumptions.
2. Machine Learning (ML) / Deep Learning (DL): Techniques like Recurrent Neural Networks (RNNs) can model high-dimensional structures but often fail to provide valid inference when data is Missing-At-Random (MAR). In longitudinal studies, drop-out is frequently driven by the outcome itself (e.g., participants stopping due to negative emotions), creating selection bias. Standard ML imputation methods (e.g., masking, mean imputation) typically assume Missing-Completely-At-Random (MCAR), leading to biased results in MAR scenarios.

Existing attempts to combine these fields (e.g., Che et al., 2018) rely on restrictive assumptions about missingness mechanisms (e.g., missing values decaying to an empirical mean) that do not hold in complex clinical scenarios.

Methodology
The authors propose the Deep Generalised Mixed Model (DGMM), a novel neural network architecture that generalizes mixed effects models to deep learning.

• Model Structure: The DGMM decomposes the prediction of longitudinal outcomes $Y_{i,k}$ into two components:

  • Fixed Effects ($\mu_1$): Modeled by a standard feed-forward neural network ($h$) taking baseline covariates ($X_i$) and time ($T_{i,k}$) as input. This captures population-level trends.
  • Random Effects ($\mu_2$): Modeled by a Variational Auto-Encoder (VAE) structure.
    • Encoder ($e$): Maps observed data ($Y_i, T_i, X_i$) to a latent space, estimating the mean and variance of a subject-specific random effect vector $b_i$.
    • Decoder ($d$): Samples from this latent distribution to generate subject-specific deviations ($\mu_2$) which are combined with the fixed effects.
  • Distribution: The model accommodates generic outcome distributions (Gaussian, Gamma, Binomial, etc.) via link functions $G_k(\cdot)$.

• Estimation:

  • Because the marginal likelihood involves an intractable integral over non-linear random effects, the authors use Variational Inference.
  • They maximize the Evidence Lower Bound (ELBO), which consists of a reconstruction term (approximated via Monte Carlo sampling) and a Kullback-Leibler (KL) divergence term regularizing the latent distribution against a standard normal prior.
  • Missing Data Handling: To address MAR data where input sizes vary, the authors employ a stochastic approximation Expectation-Maximization (EM) algorithm combined with a Robbins–Monro process.
    • Missing outcomes are imputed iteratively. In each epoch, parameters are updated via stochastic gradient descent, and missing values are simulated from the posterior predictive distribution using a Random Walk Metropolis–Hastings (RWMH) algorithm.
    • This allows the encoder to utilize the pattern of missingness for inference, rather than simply masking it.

• Prediction: Dynamic predictions for new subjects are generated by integrating over the posterior distribution of random effects and model parameters, approximated via Monte Carlo sampling.

Key Contributions

1. Architecture: The first framework to generalize mixed effects models to deep learning by estimating both fixed and random effects using neural networks, allowing for semi-parametric and flexible modeling of within- and between-person heterogeneity.
2. MAR Inference: A specific adaptation of VAEs and Bayesian data augmentation that enables valid inference under Missing-At-Random mechanisms, addressing the selection bias often ignored in standard deep learning applications for longitudinal data.
3. Generalizability: The model supports various outcome distributions and scales to high-dimensional settings, overcoming the rigidity of standard MEMs and the inference limitations of standard RNNs.

Results
The authors evaluated the DGMM on the GrowIt! study (adolescent mental health data during the pandemic) and various simulation studies.

• GrowIt! Application:

  • The DGMM was applied to predict Negative Affect, Loneliness, and Boredom.
  • Performance: The DGMM showed poorer fit on observed data compared to a standard RNN, particularly for binary outcomes.
  • Forecasting: Both models struggled to forecast future trajectories accurately. The DGMM tended to collapse to a stable linear model, while the RNN produced overly flexible but unstable forecasts.
  • Conclusion: The authors attribute the suboptimal performance to data characteristics (small sequential correlations, skewed distributions, and high missingness ~61%) rather than a fundamental flaw in the architecture, suggesting that for this specific dataset, clinical decisions should rely on observed data rather than complex forecasting.

• Simulation Studies:

  • No Missing Data: The DGMM fit well but showed slightly worse predictive performance than the RNN on independent test sets.
  • Missing Data (MAR): The DGMM demonstrated model instability, with high variance in training error and predictive performance across iterations. This was attributed to the interaction between the data imputation algorithm, variational inference, and neural network initialization.
  • Bias: Despite instability, when the DGMM converged, it recovered true trajectories with less bias than the RNN under MAR conditions, successfully leveraging the missingness mechanism.
  • Imputation: The data augmentation algorithm accurately imputed missing values in the training data, though with some variance.

Significance and Claims
The paper modestly claims that while the DGMM is a promising architecture for analyzing nonlinear hierarchical data, it currently suffers from instability issues (sensitivity to hyperparameters, posterior collapse, and variance in convergence).

• Potential: The model holds potential for providing valid inference under MAR conditions where standard ML fails, particularly as more data is collected over time.
• Limitations: The current implementation trades interpretability for flexibility. The authors note that the deep latent variable representation was treated as a byproduct in this study.
• Future Directions: The authors suggest that future iterations could improve stability using techniques like KL-annealing, $\beta$-VAE, or guaranteeing identifiability. They also propose extending the architecture to other data types (e.g., medical imaging) and utilizing the latent dimensions for unsupervised tasks (clustering) or joint modeling of time-to-event outcomes.

In summary, the DGMM represents a theoretical advancement in unifying statistical mixed models with deep learning for longitudinal data, offering a mechanism to handle MAR data, but its practical utility is currently limited by training instability and the specific characteristics of the datasets tested.