Model-based and model-free valuation signals in the human brain vary markedly in their relationship to individual differences in human behavioral control

This large-scale fMRI study reveals that while model-free valuation signals are ubiquitous in the human brain regardless of behavioral strategy, model-based signals are strictly linked to individual behavioral reliance and their absence correlates with deficits in forming accurate environmental predictions.

The Gist

The Big Picture: Two Brains in One Head

Imagine your brain is a busy city with two different navigation systems working side-by-side to help you get around:

1. The "Habit GPS" (Model-Free): This system is like a loyal, old-school taxi driver who knows the city by heart. It doesn't care about traffic maps or why a road is closed; it just remembers, "Last time I took this street, I got a pizza. Let's go there again!" It learns by trial and error. If you get a reward, it says, "Do that again." If you don't, it says, "Don't do that." It's fast, automatic, and relies on muscle memory.
2. The "Strategic Planner" (Model-Based): This system is like a sophisticated GPS app (like Google Maps) that builds a mental map of the whole city. It understands the rules: "If I turn left here, I will likely hit a dead end, but if I go right, I might find a shortcut." It simulates the future before acting. It's flexible and smart, but it takes more mental energy to run.

Usually, we use a mix of both. But this study asked a big question: Do different people rely on these systems differently, and does their brain show it?

The Experiment: A Space Mining Game

To find out, the researchers put 179 people in an MRI machine (a giant camera that takes pictures of the brain) and played a video game called "Space Miner."

• The Setup: You are a miner. You choose one of two spaceships (Yellow or Blue).
• The Twist: The Yellow ship usually goes to the Red Planet, but sometimes (rarely) it gets blown off course to the Green Planet. The Blue ship is the opposite.
• The Goal: Once you land on a planet, you have to pick a mining pad. The goal is to find gems (points). The "gem-rich" planet changes over time.

How the game tests your brain:

• If you are a Habit Driver, you just remember: "I picked Yellow and got a gem. I'll pick Yellow again." It doesn't matter if you got lucky with a rare jump to the Green Planet.
• If you are a Strategic Planner, you think: "I picked Yellow, but I ended up on the Green Planet by accident. Since the Green Planet is usually bad for gems, I should switch to Blue next time to avoid that accident."

The Discovery: Four Types of Players

The researchers looked at how 179 people played and found they naturally fell into four distinct groups:

1. The Pure Planners: They used the "Strategic Planner" logic almost exclusively.
2. The Pure Habituals: They relied almost entirely on the "Habit GPS."
3. The Mixers: They used a healthy blend of both.
4. The "Lost" Group: These people didn't seem to use either strategy effectively. They were just guessing or playing randomly.

The Brain Findings: What the MRI Showed

This is where it gets fascinating. The researchers looked at the Ventromedial Prefrontal Cortex (vmPFC), a part of the brain known as the "Value Calculator." This is where the brain decides how good an option is.

1. The Habit Signal (Model-Free) is Everywhere

The Analogy: Imagine a radio playing a song in the background.

• The Finding: The "Habit GPS" signal was playing loudly in the brains of everyone, even the people who were playing the game using pure logic (The Planners) or even the "Lost" group.
• What it means: Your brain is always calculating "If I do this, I get a reward," even if you aren't consciously using that information to make your choice. It's a default setting that never turns off. It's like a background hum that is always there, regardless of whether you are driving the car or just sitting in the passenger seat.

2. The Planner Signal (Model-Based) is Optional

The Analogy: Imagine a spotlight that only turns on when you need it.

• The Finding: The "Strategic Planner" signal was only bright and active in the brains of the people who were actually using that strategy.
  • If you were a Pure Planner, the spotlight was blazing.
  • If you were a Pure Habitual, the spotlight was off.
  • If you were in the "Lost" group, the spotlight was also off.
• What it means: The brain doesn't waste energy building a complex mental map unless you are actually using it to make decisions. If you aren't planning ahead, your brain doesn't bother lighting up the "planning" department.

3. Why are some people "Lost"?

The researchers found something surprising about the "Lost" group. They didn't just lack the "Planner" signal; they also lacked the State Prediction Error signal.

• The Analogy: Imagine a GPS that tries to update its map when it sees a new road, but the update fails.
• The Finding: The "Lost" group's brains struggled to update their mental map of how the spaceships moved between planets. They couldn't accurately predict, "If I pick Yellow, I will likely go to Red."
• The Conclusion: It seems that some people struggle to build the internal "map" of the world in the first place. Because they can't build the map, they can't use the "Strategic Planner" strategy. They are stuck relying on the "Habit GPS" or guessing, because their brain's ability to simulate the future is broken.

The Takeaway

This study tells us that human brains are diverse:

1. The "Habit" system is universal: Everyone's brain calculates rewards automatically, whether they like it or not. It's the background noise of our decision-making.
2. The "Planning" system is selective: We only light up our "planning" brain centers when we actually use that strategy.
3. The root of the difference: The reason some people don't use the "planning" strategy isn't just a preference; for some, it might be because their brains have a harder time building the mental map required to plan in the first place.

In short, we are all driving with a "Habit GPS" always on, but only some of us are actively using the "Strategic Planner" to navigate the complex roads of life. And for a few, the map itself is hard to read.

Technical Summary

1. Problem Statement

Human decision-making under reinforcement is widely theorized to rely on two distinct computational strategies:

• Model-Free (MF) Reinforcement Learning: A habitual system that learns action values through trial-and-error accumulation of rewards without an internal model of the environment.
• Model-Based (MB) Reinforcement Learning: A goal-directed system that utilizes an internal cognitive model of state transitions to plan and update values flexibly.

While average human behavior often reflects a mixture of both strategies, the neural basis of individual differences in strategy expression remains unclear. Specifically, it is unknown whether:

1. Brain regions (like the ventromedial prefrontal cortex, vmPFC) flexibly encode whichever strategy an individual is currently using.
2. Distinct neural mechanisms are recruited exclusively for MB or MF strategies depending on the individual's behavioral profile.
3. Individuals who do not exhibit MB behavior at the behavioral level also lack the corresponding neural signals, or if these signals exist but are not utilized.

2. Methodology

Participants and Design

• Sample: A large-scale fMRI study with 179 participants (N=179), an unusually large sample for task-based fMRI, recruited from the general Los Angeles population.
• Task: A variant of the Two-Step Task (Daw et al., 2011). Participants chose between two "spaceships" (Stage 1) leading probabilistically (70% common, 30% rare) to two "planets" (Stage 2), which then led to landing pads and finally a reward outcome.
• Manipulations: The task included dynamic reward probability shifts, reward magnitude variations, and state-transition uncertainty manipulations to dissociate MF and MB learning signals.

Behavioral Analysis and Clustering

• Clustering: Instead of treating the population as a homogeneous mix, the authors applied k-means clustering to behavioral features (stay-switch probabilities and reaction times) derived from a larger external dataset (N=678).
• Groups Identified: Participants were assigned to four distinct behavioral clusters:
  1. MB Group: Predominantly model-based.
  2. MF Group: Predominantly model-free.
  3. Mixture Group: Utilizes both strategies.
  4. Other Group: Does not rely on standard RL mechanisms (blunted or non-RL behavior).
• Computational Modeling: An Arbitration Mixture Model (a hybrid RL model with separate MF and MB modules and dynamic weighting) was fitted to individual data to extract computational variables (e.g., $w_{MF}$, value signals, prediction errors).

Neuroimaging Analysis

• fMRI Acquisition: 3T Siemens Prisma scanner; standard preprocessing (fMRIPrep).
• General Linear Model (GLM): First-level GLMs included parametric modulators for:
  • Decision Values: Chosen vs. Rejected values for both MF and MB systems.
  • Prediction Errors: State Prediction Errors (SPE) and Reward Prediction Errors (RPE) for both MF and MB systems.
• Second-Level Analysis:
  • Group-level analyses (one-sample t-tests) to identify general neural correlates.
  • Parametric modulation by individual behavioral weights ($w_{MF}$) to test for correlations between neural signal strength and behavioral strategy.
  • Region of Interest (ROI): Primary focus on vmPFC (based on meta-analytic masks) and striatum (for RPE), with whole-brain exploratory analyses.

3. Key Contributions

1. Large-Scale Individual Difference Analysis: By utilizing a large sample (N=179) and clustering participants into distinct behavioral phenotypes, the study moves beyond population averages to map neural mechanisms to specific behavioral strategies.
2. Dissociation of Neural Encoding: The study provides empirical evidence that Model-Based value signals are behaviorally contingent, whereas Model-Free value signals are ubiquitous across individuals regardless of their behavioral strategy.
3. Mechanism for MB Deficits: The study links the absence of MB behavioral control to a failure in State Prediction Error (SPE) encoding, suggesting that individuals who do not use MB strategies may lack the ability to accurately learn or update the internal model of the environment.

4. Key Results

A. Behavioral Findings

• The population exhibited a mix of strategies, but individual clustering revealed distinct groups (MB, MF, Mixture, Other).
• The MB Group showed strong sensitivity to transition types (rare vs. common) when making choices, consistent with MB control.
• The MF Group showed strong sensitivity to previous rewards regardless of transition type.
• The Other Group showed blunted reward sensitivity and poor task performance, suggesting a lack of effective RL strategy.

B. Neural Correlates of Decision Values

• Model-Based (MB) Values:
  • vmPFC Encoding: MB decision value signals in the vmPFC were strongly correlated with the degree of MB behavioral reliance.
  • Group Differences: Significant MB value signals were found only in the MB group. The Mixture group showed modest effects, while the MF and Other groups showed no significant MB value signals in the vmPFC.
  • Conclusion: MB representations appear only when the individual actively utilizes the MB strategy.
• Model-Free (MF) Values:
  • Ubiquity: MF decision value signals were significantly present in the vmPFC across all groups, including the MB group and the "Other" (non-RL) group.
  • Independence: The strength of MF signals in the vmPFC did not correlate with the degree of MF behavioral reliance ($w_{MF}$). Even individuals who behaved purely MB showed robust MF value signals in the brain.
  • Conclusion: MF value computation appears to be a baseline default process in the brain, occurring independently of whether it is used to drive behavior.

C. Prediction Error Signals (Learning Mechanisms)

• State Prediction Errors (SPE): SPE signals (crucial for learning the transition structure in MB systems) were found in the dorsolateral prefrontal cortex (dlPFC) and intraparietal sulcus (IPS).
• Correlation with Behavior: The magnitude of SPE encoding was negatively correlated with $w_{MF}$ (i.e., stronger SPE in MB users).
• The "Other" Group: While the "Other" group showed some SPE signals in ROI analyses, whole-brain voxel-wise analysis showed no significant SPE clusters in this group. This suggests that the lack of MB behavior in these individuals may stem from an inability to encode or update the internal state-transition model (SPE), rather than just a failure to use the resulting value.

D. Reward Prediction Errors (RPE)

• RPE signals were observed in the striatum, but due to high collinearity between MF and MB RPE signals, distinct separation was difficult. However, the study confirmed the presence of these signals in the expected striatal regions.

5. Significance and Implications

• Asymmetry in Neural Implementation: The findings reveal a fundamental asymmetry: MB control is a "switched-on" system that requires specific neural engagement (vmPFC, dlPFC, IPS) and fails if the internal model cannot be learned (SPE deficits). In contrast, MF control is a "always-on" background process that generates value signals regardless of behavioral output.
• Clinical Relevance: This framework helps explain individual differences in psychiatric conditions (e.g., OCD, addiction, depression) where MB control is often impaired. The impairment may not just be a failure of decision-making but a failure in the formation of the internal world model (SPE encoding).
• Methodological Advancement: The study demonstrates the necessity of large-sample fMRI studies combined with computational clustering to resolve heterogeneity in human decision-making, moving beyond the "average brain" to understand distinct neural phenotypes.

In summary, the paper establishes that while the brain constantly computes model-free values, the ability to utilize model-based planning is a distinct, capacity-limited process dependent on the successful encoding of state transitions, which varies significantly across individuals.