🤖 machine learning

DTVEM-RE: A Hierarchical Random-Effects Extension of the Differential Time-Varying Effect Model for Person-Specific Multi-Lag Estimation in Intensive Longitudinal Data

This paper introduces DTVEM-RE, a hierarchical random-effects extension of the Differential Time-Varying Effect Model that enables person-specific multi-lag estimation in intensive longitudinal data through both discrete-time Bayesian VAR and continuous-time Ornstein-Uhlenbeck frameworks, demonstrating superior predictive accuracy and the ability to capture individual differences in lag structures that traditional methods miss.

Original authors: Amartya Bhattacharya

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amartya Bhattacharya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how people's moods change throughout the day. You have a group of 40 people who text you their feelings four times a day for a month. You want to know: Does how someone feels right now predict how they will feel an hour later? Or maybe two hours later?

This is the problem the paper tackles. It introduces a new tool called DTVEM-RE. To understand why it's special, let's look at the old way of doing things versus the new way.

The Old Way: The "One-Size-Fits-All" Recipe

The original tool (called DTVEM) works like a master chef who tastes a giant pot of soup made by mixing everyone's ingredients together.

The Taste Test: The chef tastes the "average" soup to figure out the best recipe. They might say, "Okay, for this group, the flavor of sadness today predicts sadness tomorrow."
The Problem: The chef then applies that exact same recipe to every single person. They assume that because the group average works, it works for everyone.
The Flaw: In reality, some people are like spicy food lovers (their moods change fast), while others are like bland food lovers (their moods stay the same for a long time). By using one recipe for everyone, the old tool misses these huge differences. It's like telling a marathon runner and a sprinter to run at the exact same pace because "that's what the average person does."

The New Way: DTVEM-RE (The "Personalized Tailor")

The authors created DTVEM-RE (Random Effects). Think of this not as a master chef, but as a tailor who makes custom suits.

The Group Pattern (The Fabric): First, the tailor looks at the whole group to see the general style of the fabric. This is the "average" pattern.
The Personal Fit (The Custom Cut): Then, the tailor measures each individual to see how their specific body differs from the average.
- If Person A is very different from the average, the tailor adjusts the suit significantly for them.
- If Person B is very similar to the average, the tailor makes only tiny adjustments.
- The Magic: This process uses a statistical "shrinkage" trick. It pulls extreme individual guesses toward the group average if there isn't enough data for that specific person, but lets them stand out if the data is strong. This prevents wild guesses while still capturing unique traits.

What Did They Find?

The paper tested this new tailor on real data from people with anxiety and depression. Here are the three main discoveries, explained simply:

1. Everyone is Different (and we knew it, but now we can measure it)
The old tool said, "Everyone has a lag of 1 hour." The new tool said, "Actually, for some people, their mood today predicts tomorrow (a long lag), while for others, their mood changes so fast that today doesn't predict tomorrow at all."

The Result: The differences between people were huge. Some people's moods were almost completely independent of the past, while others were very sticky and persistent. The new tool successfully measured these differences without getting confused by the noise.

2. The "Sweet Spot" Changes for Everyone
This is the most surprising part. The researchers looked at not just "1 hour later," but also "2 hours later" and "3 hours later."

The Discovery: For some people (specifically those tracking "worry"), the biggest difference between individuals showed up at 3 steps later. For others (tracking "energy"), the biggest difference showed up at 2 steps later.
The Analogy: Imagine a group of musicians. The old tool assumed everyone's instrument was out of tune at the same note. The new tool found that for the violinists, the problem was at the high note, but for the drummers, the problem was at the low note. Previous tools could only check the "high note" for everyone; this new tool checks all notes for everyone.

3. It's Better at Predicting the Future
The authors tested if this new method could guess the next mood rating better than the old methods.

The Result: It did slightly better. It wasn't a miracle cure that predicted the future perfectly, but it was the most accurate of the bunch. More importantly, it gave a better "confidence score" (it knew when it was unsure).

Why Does This Matter?

The paper argues that in psychology, we often assume everyone's brain works the same way on a group level. But this is like assuming all cars have the same engine.

DTVEM-RE allows researchers to say: "We know the average car engine, but here is the specific engine map for this driver."
It confirms that the "time lag" (how long it takes for a feeling to fade or build up) is a unique personal trait, not a universal rule.

What the Paper Doesn't Say

The authors are very careful to say what they haven't done yet:

They haven't proven that this will change how doctors treat patients today.
They haven't said that "Person A needs a different drug" based on this.
They haven't tested if these patterns stay the same for a person over years (reliability).

They are simply saying: "We built a better measuring tape that shows everyone is different in specific, measurable ways, and previous tools were too blunt to see it."

Summary

Think of DTVEM-RE as upgrading from a group photo (where everyone is blurred together) to a high-definition portrait of every single person, while still keeping the context of the whole room. It proves that when it comes to mood and anxiety, the "average" person is a myth, and the real story is found in the unique, individual differences.

Technical Summary: DTVEM-RE

Problem Statement
The Differential Time-Varying Effect Model (DTVEM), introduced by Jacobson, Chow, and Newman (2019), is a standard tool for identifying optimal lag structures in intensive longitudinal data (ILD). It employs a two-stage hybrid procedure: an exploratory Generalized Additive Mixed Model (GAMM) to smooth autoregressive coefficients across lag distances, followed by a confirmatory State-Space Vector Autoregression (VAR). However, the published DTVEM assumes a single group-level lag structure shared across all individuals. This assumption conflicts with the idiographic premise of modern psychopathology research, which posits that individuals differ meaningfully in their psychological dynamics. As noted by the original authors, the lack of person-specific lag estimation is a critical limitation that prevents the method from capturing between-person heterogeneity in dynamic processes.

Methodology
The paper introduces DTVEM-RE (DTVEM with Random Effects), a hierarchical extension that estimates person-specific multi-lag coefficients with shrinkage toward group-level means. The framework consists of two stages:

Stage 1: Hierarchical Exploratory GAMM
- Extends the standard DTVEM Stage 1 by fitting a factor-smooth GAMM using the mgcv::bam package.
- Models the lag-effect curve as a sum of a population-level smooth $f(\Delta t)$ and person-specific deviations $b_i(\Delta t)$ .
- Uses a factor-smooth basis (bs = "fs") to couple all participant-specific smooths through a single shared smoothing parameter. This provides automatic per-person regularization, shrinking individual curves toward the population mean where data is sparse and allowing distinct shapes where data is informative.
Stage 2: Hierarchical Bayesian State-Space VAR
- Implements a confirmatory VAR model estimated via Hamiltonian Monte Carlo (HMC) in Stan.
- Univariate Case: For a single item, person-specific autoregressive coefficients ( $a_i$ $a_{i}$ ) and residual standard deviations ( $\sigma_i$ $σ_{i}$ ) are modeled hierarchically:
  - $a_i \sim N(\mu_a, \tau_a^2)$ , where $\mu_a$ is the population mean and $\tau_a$ is the between-person standard deviation.
  - $\sigma_i$ follows a hierarchical log-normal distribution to ensure positivity and handle scale parameters.
- Multi-Lag Extension: The model generalizes to $K$ lags by assigning separate hyperparameters ( $\mu_k, \tau_k$ ) to each lag. This allows the magnitude of between-person heterogeneity to vary non-monotonically across lags, a feature not supported by existing methods that restrict random effects to lag-1.
- Parameterization: Uses non-centered parameterizations for person-specific coefficients and residual SDs to avoid funnel-shaped posterior pathologies and ensure efficient sampling (zero divergent transitions).
- Priors: Weakly informative priors are used (e.g., $N(0, 0.5)$ for means, Half-Normal(0, 0.3) for between-person SDs) to express skepticism toward implausibly large heterogeneity while covering the stationary range.

Key Contributions

Model Specification & Implementation: The authors specify the DTVEM-RE model and provide a full Stan implementation with code released for replication. It retains standard DTVEM as a fixed-effects special case (when $\tau = 0$ ).
Parameter Recovery Validation: A simulation study ( $N=40, T=130$ ) across four heterogeneity levels confirms that the between-person variance parameter ( $\tau_a$ ) is recovered with absolute bias below 0.01 and credible interval coverage of 90–93%, validating the method's reliability at typical Ecological Momentary Assessment (EMA) sample sizes.
Empirical Demonstration: Applied to the Fisher et al. (2017) dataset ( $N=40$ $N = 40$ outpatients with GAD/MDD), the method demonstrates:
- Substantial person-specific heterogeneity in lag-1 effects (spanning an order of magnitude).
- High agreement ( $r=0.87$ to $0.92$) between hierarchical Bayesian estimates and independent per-person GAMM estimates.
- Superior one-step-ahead predictive performance (log-likelihood and RMSE) compared to group-only, naive per-person, and simple hierarchical baselines, albeit by modest margins.
Multi-Lag Heterogeneity Discovery: The multi-lag extension reveals that between-person heterogeneity ( $\tau_k$ ) is statistically robust (90% CI excludes zero) at lags 1, 2, and 3 simultaneously. Crucially, the lag at which heterogeneity is largest differs by affect item (e.g., lag 1 for "down," lag 2 for "energetic," lag 3 for "worried"), a pattern undetectable by methods restricting random effects to lag-1.

Results

Simulation: The between-person SD ( $\tau_a$ ) was recovered without meaningful bias. The population mean ( $\mu_a$ ) showed a small, consistent downward bias (~0.02) consistent with finite-sample AR(1) estimation theory, but this did not affect the estimation of heterogeneity.
Empirical Lag-1: All three affect items ("down," "worried," "energetic") showed significant between-person variation in lag-1 coefficients. The hierarchical model provided calibrated uncertainty quantification, with 39 of 40 participants showing non-zero autoregressive dependence.
Empirical Multi-Lag: All nine estimates of between-person SD across three items and three lags were significantly greater than zero. The shape of heterogeneity was non-monotonic and item-specific, contrasting with the monotonic decay of population means.
Prediction: DTVEM-RE achieved the best predictive log-likelihood and lowest RMSE among four baselines. However, the absolute gain was modest (approx. 0.01 log-likelihood units per beep), attributed to the stability of naive per-person OLS at $T \approx 110$ . The primary value in this regime is identified as calibrated uncertainty and multi-lag capability rather than dramatic predictive superiority.

Significance and Claims
The paper claims that DTVEM-RE provides, to the authors' knowledge, the first principled idiographic implementation of DTVEM-style lag detection. It directly addresses the limitation identified by the original DTVEM authors by enabling the estimation of person-specific multi-lag profiles while retaining the benefits of hierarchical pooling.

The authors emphasize that the methodological novelty lies in combining nonparametric multi-lag exploration with hierarchical pooling on person-specific coefficients at each detected lag. This allows for the detection of complex heterogeneity structures (e.g., varying peaks of heterogeneity across lags) that existing hierarchical VAR methods (like mlVAR) cannot capture.

The paper remains modest regarding clinical implications, stating that the results are exploratory and do not license direct clinical recommendations. The authors frame the finding of multi-lag heterogeneity as a candidate phenomenon requiring replication in larger, diverse samples and further investigation into stability and clinical relevance, rather than as immediate evidence for changing clinical practice. The method is presented as a tool for describing person-specific lag structure, with the clinical leverage of such descriptions remaining a separate empirical question.