📊 statistics

Joint modeling for learning decision-making dynamics in behavioral experiments

This paper proposes a novel joint modeling framework that integrates reinforcement learning and drift-diffusion models with hidden Markov switching to analyze reward-based decision-making, demonstrating through numerical studies and EMBARC data that this approach effectively captures strategy alternation and reveals distinct brain-behavior associations specific to the "engaged" state in Major Depressive Disorder patients.

Original authors: Yuan Bian, Xingche Guo, Yuanjia Wang

Published 2026-02-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuan Bian, Xingche Guo, Yuanjia Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a group of people play a video game where they have to guess which of two cartoon faces is the "winner" to get points. Sometimes they get it right and get a reward; sometimes they don't.

For a long time, scientists trying to understand how people learn from these games used two different tools:

The Scorekeeper: This tool tracks how people change their guesses based on past rewards (like learning that "Face A" usually wins).
The Stopwatch: This tool measures how long it takes to make a guess, assuming that a quick guess might mean a lucky guess, while a slow guess means careful thinking.

The Problem:
Most people don't play the game perfectly the whole time. Sometimes they are fully focused (engaged), thinking hard and learning from every point. Other times, they zone out, get tired, or get distracted (lapsed), and just start guessing randomly without caring about the score.

If you only look at the score or only look at the stopwatch, you miss the big picture. You might think a slow guess means "smart thinking," when actually, it might just mean the person was confused or distracted.

The New Solution: The "Smart Coach" Framework
The authors of this paper built a new, super-smart "coach" (a mathematical model) that does three things at once:

It learns: It tracks how the player updates their strategy based on rewards (Reinforcement Learning).
It times: It measures how long the player takes to decide, treating time as a clue about how much evidence they gathered (Drift-Diffusion Model).
It detects mood: It acts like a detective, figuring out if the player is currently in "Engaged Mode" (thinking hard) or "Lapsed Mode" (zoning out/guessing randomly).

How It Works (The Analogy):
Think of the player's brain as a car with two gears:

Gear 1 (Engaged): The engine is running. The driver is looking at the road, adjusting speed based on traffic (rewards), and taking time to make safe turns. This is the "smart" part.
Gear 2 (Lapsed): The driver has taken their hands off the wheel. The car is just rolling forward randomly. The driver isn't looking at the road; they are just guessing where to go.

The new model doesn't just watch the car; it knows which gear the driver is in at every single moment. It realizes that when the car is in "Lapsed Mode," the time it takes to turn doesn't mean the driver is thinking hard; it just means they are drifting.

What They Found (The Real-World Test)
The researchers tested this "Smart Coach" on data from a real study involving people with Major Depressive Disorder (MDD) and healthy people.

Here is what they discovered:

Healthy People: They spent most of their time in "Engaged Mode." When they were engaged, they made good decisions. When they slipped into "Lapsed Mode," they guessed randomly.
People with Depression: They spent less time in "Engaged Mode." They were more likely to be in "Lapsed Mode" (zoning out).
The Speed Difference: When people with depression did get into "Engaged Mode," they took longer to make a decision than healthy people. It was like they were trying harder to focus, but it took more effort.
The Brain Connection: When they looked at brain scans, they found that brain activity was linked to how people behaved only when they were in "Engaged Mode." When they were in "Lapsed Mode," their brain activity didn't match their behavior at all.

Why This Matters
This is a big deal because it changes how we understand depression. It's not just that depressed people are "bad at making decisions." It's that they struggle to stay engaged in the task.

By separating the "zoning out" moments from the "thinking" moments, scientists can finally see the true link between the brain and behavior. It's like trying to hear a conversation in a noisy room: this new method helps them turn down the noise (the random guessing) so they can clearly hear the conversation (the actual decision-making process).

In a Nutshell:
This paper gives us a better way to watch how people think. It tells us that to understand the brain, we have to know when the person is actually paying attention and when they are just daydreaming. For people with depression, the problem isn't just how they think, but how often they can get their brain to start thinking in the first place.

1. Problem Statement

Major Depressive Disorder (MDD) is associated with cognitive deficits, concentration issues, and abnormalities in reward processing. While individuals with MDD often exhibit slower decision-making compared to healthy controls, existing computational models fail to fully capture the complexity of these behaviors.

Limitations of Reinforcement Learning (RL): Standard RL models effectively capture how behavior adapts across trials based on reward prediction errors but treat decisions as static probabilities (e.g., softmax functions). They ignore the temporal dynamics and speed-accuracy trade-offs inherent in decision-making.
Limitations of Drift-Diffusion Models (DDM): DDMs successfully model the continuous accumulation of evidence and response times but typically assume a static decision process that does not evolve through learning over trials.
The Strategy Switching Gap: Both RL and DDM generally assume a consistent decision-making strategy. However, empirical evidence suggests that individuals alternate between an "engaged" state (deliberate, reward-driven decision-making) and a "lapsed" state (random guessing or inattention). Ignoring these latent state switches leads to biased parameter estimates and incomplete behavioral characterizations.

Objective: To develop a unified framework that jointly models reward-based learning, response times, and latent strategy switching to better characterize decision-making dynamics in MDD patients versus healthy controls.

2. Methodology: The RL-HMM-DDM Framework

The authors propose a novel RL-HMM-DDM framework that integrates three components:

A. Core Components

Reinforcement Learning (RL): Models the evolution of expected values ( $Q$ ) using a Rescorla-Wagner update rule based on reward prediction errors. This captures how subjects learn from trial outcomes.
Drift-Diffusion Model (DDM): Models the decision process as a continuous accumulation of noisy evidence ( $W(t)$ $W (t)$ ) until it hits an upper ( $\alpha$ $α$ ) or lower ($0$) boundary.
- Drift Rate ( $v$ ): In the "engaged" state, the drift rate is linearly coupled to the reward contrast ( $Z$ ) derived from the RL component ( $v = c \cdot Z$ ). This links learning to the speed of evidence accumulation.
- Non-decision Time ( $\tau$ ): Accounts for perceptual encoding and motor execution.
Hidden Markov Model (HMM): Introduces a latent binary variable $U_{i,j} \in \{0, 1\}$ $U_{i, j} \in {0, 1}$ representing the strategy at trial $j$ $j$ for subject $i$ $i$ .
- State 1 (Engaged): Decisions follow the full RL-DDM. The subject is actively learning and processing rewards.
- State 0 (Lapsed): Decisions follow a simplified DDM where the drift rate is fixed at 0 and initial bias at 0.5. This approximates random guessing with equal probability, independent of stimuli.
- Transition Probabilities: Modeled via a logistic regression where transition probabilities between states depend on subject-level covariates (e.g., MDD status).

B. Parameter Estimation

Algorithm: A computationally efficient Generalized Expectation-Maximization (EM) algorithm is used.
E-Step: Utilizes the Forward-Backward algorithm to compute posterior probabilities of the latent states ( $U_{i,j}$ ) and state transitions, given the observed actions and response times.
M-Step: Updates model parameters (learning rate $\beta$ , boundary $\alpha$ , drift scaling $c$ , non-decision time $\tau$ , and HMM transition parameters) by maximizing the expected complete-data log-likelihood.

C. Post-Estimation Metrics

The framework generates individual-level metrics:

Engagement Probability: The posterior probability of being in the "engaged" state at any trial.
Engagement Score: A logit-transformed average of engagement probabilities across trials.
State-Specific Response Times: Separate mean response times for engaged and lapsed trials.

3. Key Contributions

Unified Framework: First to jointly integrate RL (learning), DDM (temporal dynamics), and HMM (latent strategy switching) into a single probabilistic model.
Handling Latent Heterogeneity: Explicitly models the alternation between "engaged" and "lapsed" strategies, addressing the issue where standard mixture models fail to identify parameters reliably without response time data.
Covariate-Dependent Transitions: Extends previous HMM-RL work by allowing transition probabilities between states to depend on covariates (e.g., clinical diagnosis), rather than assuming constant transition rates.
Robust Estimation: Demonstrates that incorporating response times significantly improves the accuracy of identifying latent states compared to methods that only use choice data.

4. Results

A. Simulation Studies

Setup: 200 simulations across varying sample sizes ( $N=100, 200$ ) and trial counts ( $J=100, 200$ ), with both binary and continuous reward distributions.
Performance with Switching: In scenarios with strategy switching, the proposed RL-HMM-DDM significantly outperformed standalone RL-DDM and RL-HMM methods.
- RL-DDM (assuming no switching) severely underestimated parameters.
- RL-HMM (ignoring response times) showed bias in learning rate and transition parameters.
- RL-HMM-DDM achieved high accuracy (>80%) in recovering latent states and decision actions, whereas competitors achieved ~65-80%.
Performance without Switching: When no switching occurred, RL-HMM-DDM performed comparably to the standard RL-DDM, demonstrating robustness.
Robustness: The method remained robust under input perturbations (noise in actions or response times).

B. Application to EMBARC Study

The model was applied to the Probabilistic Reward Task (PRT) data from the EMBARC study ( $N=184$ : 153 MDD, 31 Controls).

Engagement Rates: MDD patients exhibited lower overall engagement compared to healthy controls.
Response Times:
- Engaged State: MDD patients took significantly longer to make decisions than controls when engaged.
- Lapsed State: Both groups showed similarly short decision times (random guessing), indicating the "lapsed" state is distinct from the slowed processing in the engaged state.
Learning: Both groups showed decreasing response times for the "rich" reward over trials, indicating reinforcement learning, but MDD patients remained slower overall.
Brain-Behavior Associations:
- Engaged State: Engagement scores and engaged response times were significantly associated with fMRI (connectivity in dACC, PCC, dlPFC, and insula) and behavioral phenotyping (Flanker task performance).
- Lapsed State: Response times in the lapsed state showed no significant association with any brain measures.
- Significance: This provides evidence that brain-behavior associations in MDD are specific to the "engaged" cognitive state, suggesting that neural mechanisms of attention and cognitive control are disrupted specifically during deliberate decision-making.

5. Significance

Clinical Insight: The framework reveals that MDD is characterized not just by slower processing, but by a reduced propensity to enter a high-engagement cognitive state. When they do engage, their processing is inefficient (slower).
Methodological Advancement: By distinguishing between "engaged" and "lapsed" states, the model prevents the conflation of attentional lapses with genuine cognitive deficits, leading to more accurate biomarkers.
Neuroscience Implications: The finding that brain-behavior associations exist only in the "engaged" state highlights the importance of state-dependent analysis in neuroimaging. It suggests that neural circuits (PFC-ACC, insula-ACC) modulate cognitive effort specifically during active decision-making, a mechanism that is compromised in MDD.
Future Directions: The framework is extensible to non-linear RL-DDM integrations, multi-action tasks (race models), and joint modeling of multiple behavioral modalities.

In summary, this paper presents a rigorous statistical framework that bridges the gap between learning theory and cognitive dynamics, offering a powerful tool to dissect the complex behavioral and neural signatures of psychiatric disorders like MDD.