🧬 biology

Beyond model-based and model-free learning: irrational learning of successor features in addictive behaviour

This study challenges the traditional model-based versus model-free dichotomy in addiction research by demonstrating that individuals with substance use disorders exhibit irrational learning characterized by a resource-rational shift toward simplified, reusable successor features to minimize cognitive costs, even at the expense of reward maximization.

Original authors: Ru-Yuan Zhang, Zeming Fang, Yu-Yan Gao, Yu-Feng Xia, Zi-Jian Cheng, Lingxiao Yang, Wei Li, Tomoko Kishimoto, Lei Guo, Trevor Robbins

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ru-Yuan Zhang, Zeming Fang, Yu-Yan Gao, Yu-Feng Xia, Zi-Jian Cheng, Lingxiao Yang, Wei Li, Tomoko Kishimoto, Lei Guo, Trevor Robbins

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine your brain as a super-smart video game console that has to play thousands of different games at once. Some games are simple, like a game of "Rock, Paper, Scissors" where you just memorize that "Rock beats Scissors." Other games are complex strategy adventures where you have to map out the whole world, predict how every enemy will move, and plan ten steps ahead. For a long time, scientists thought our brains were stuck in a tug-of-war between these two styles: the "lazy" style that just memorizes what worked before (Model-Free), and the "hardworking" style that builds a mental map of the world to plan ahead (Model-Based). They believed that in addiction, the brain gets lazy and stops planning, relying only on old habits. But what if the brain isn't just lazy or hardworking? What if it's actually trying to be a genius accountant, constantly calculating how much "brain power" (or cognitive effort) a decision costs versus how much "fun" (reward) it gives back? This is the world of "resource-rationality," a fancy way of saying our brains are always trying to get the best deal for the least amount of mental energy.

Now, let's step into a new study by researchers Ru-Yuan Zhang and their team, who decided to test this idea on a group of people with substance use disorders (SUD) and a group of healthy volunteers. They didn't just ask them to play one game; they threw them into a massive, multi-level dungeon crawler. Imagine a castle with a tree-shaped map. In each round, the "market prices" for three resources—wood, stone, and iron—change. Your goal is to pick a path through the castle to grab the most valuable resources based on those prices. The twist? The prices change every time, creating four different "tasks" or games within the same castle. To win big, you have to learn the layout of the castle and quickly adapt your path whenever the prices change.

The researchers found something surprising that breaks the old "lazy vs. hardworking" rule. When they watched how people played, they saw that neither the "lazy" habit-based style nor the "hardworking" map-building style could fully explain what was happening. Instead, everyone seemed to be using a clever middle-ground strategy called "Successor Features." Think of this like learning the "vibe" of a room rather than memorizing every single step. You learn that "Room A usually leads to a pile of wood" and "Room B usually leads to stone," and you can mix and match these "vibes" to solve new price puzzles without having to rebuild your entire mental map from scratch.

However, the story takes a turn when looking at the group with addiction. While healthy players used these "vibes" to flexibly switch strategies and find the best path for every new price list, the players with addiction started to get stuck. They found the mental effort required to switch strategies too expensive. So, they adopted a "one-size-fits-all" approach. No matter what the prices were, they kept running to the same room (Room 4) because it was the easiest path to remember, even if it meant they lost out on a lot of money. The study suggests this wasn't because they couldn't learn the map, but because their brains were hyper-sensitive to the "cost" of thinking. They were willing to sacrifice rewards just to save mental energy.

The researchers built a computer model that acted like a "brain accountant," weighing the reward against the mental effort. This model, which they called the Resource-Rational Successor Feature (RRSF) model, was the only one that could perfectly predict the behavior of both groups. It showed that the addiction group had a much higher "cost sensitivity"—they felt the mental effort of switching strategies was a huge burden, so they stuck to a simple, default plan. The study also found that the more severe a person's addiction symptoms were, the more they tended to avoid mental effort and stick to their rigid, suboptimal paths.

In short, this paper suggests that addiction isn't just about a broken "planning" system or a broken "habit" system. Instead, it's about a brain that has become too frugal with its mental energy. It's like a gamer who, instead of learning the new levels, just keeps pressing the same button over and over because the new levels feel too exhausting to figure out. The study measured these behaviors in 83 people with substance use disorders and 45 healthy controls, finding that the "lazy" behavior was actually a calculated, albeit maladaptive, attempt to save cognitive effort. This new perspective helps us understand that the rigid, repetitive behaviors seen in addiction might be the brain's way of trying to protect itself from being overwhelmed, even if it means missing out on the best rewards.

Technical Summary: Beyond Model-Based and Model-Free Learning: Irrational Learning of Successor Features in Addictive Behaviour

Problem Statement
Addiction is frequently conceptualized within reinforcement learning (RL) frameworks as a shift from flexible, goal-directed "model-based" (MB) control to rigid, habitual "model-free" (MF) strategies. However, this MB–MF dichotomy is largely derived from single-task settings and may fail to capture learning dynamics in complex, multi-task environments where generalization is required. Furthermore, existing addiction models often infer cognitive effort indirectly from performance rather than quantifying the computational costs of learning itself. This study addresses the gap in understanding how individuals with Substance Use Disorders (SUD) learn and generalize across multiple tasks, specifically investigating whether their behavior is better explained by a shift to MF control or by a resource-rational adaptation involving "successor features" (SF) and cognitive effort constraints.

Methodology
The researchers recruited 83 males with SUD and 45 demographically matched healthy controls (HC). Participants performed a multi-task reinforcement learning paradigm involving a two-step decision tree.

Task Structure: In each trial, participants observed a "price set" (defining the task $g$ ) for three resources (wood, stone, iron). They then made two sequential choices to navigate a tree of rooms, ultimately reaching a final room containing fixed quantities of resources. The reward was calculated as the dot product of the resource quantities and the current price set.
Multi-Task Design: The experiment utilized four distinct price sets (tasks), categorized into two types (A and B) and two difficulty levels (easy and hard). The tasks were symmetric in potential reward but differed in the complexity of the required policy (e.g., Type B tasks favored a simple "left-left" sequence, while Type A required a "right-left" sequence).
Computational Modeling: The authors developed and compared eight computational models across two frameworks:
1. Classical RL: Model-Free (MF), Model-Based (MB), and Successor Feature (SF) models. Variants with "perseveration" (action repetition) were also tested.
2. Resource-Rational RL: These models (RRMF, RRMB, RRSF) incorporated an information-theoretic cost of cognitive effort. The objective function balanced expected reward against the Kullback-Leibler (KL) divergence between a task-specific policy and a default policy, formalized as $I_\pi(G; A|S)$ . A cost sensitivity parameter ( $\lambda$ ) controlled the trade-off between reward maximization and effort conservation.
Analysis: Models were fitted to trial-by-trial data using Maximum A Posteriori (MAP) estimation. Model comparison was conducted using Bayesian Information Criterion (BIC) and Protected Exceedance Probability (PXP). Parameter recovery and model recovery analyses were performed to validate the identifiability of the models.

Key Results

Behavioral Deficits in SUD: The SUD group earned significantly fewer rewards than the HC group and exhibited slower learning curves. Crucially, SUD participants showed reduced policy flexibility: they disproportionately selected Room 4 (the optimal room for Type B tasks) across all task types, including Type A tasks where it was suboptimal. In contrast, HC participants adapted their policies to the specific task requirements.
Failure of Classical RL Models: Standard MB, MF, and SF models failed to capture key behavioral signatures, particularly the SUD group's tendency to reuse a fixed policy across tasks and the asymmetry in performance between Type A and Type B tasks.
Superiority of Resource-Rational SF (RRSF): The Resource-Rational Successor Feature (RRSF) model provided the best fit to the data. It successfully reproduced the observed behavioral patterns, including the SUD group's reliance on a default policy (Room 4) even when suboptimal.
Mechanism of Irrationality: The RRSF model revealed that both groups utilized SF learning but differed in their sensitivity to cognitive effort ( $\lambda$ ). The SUD group exhibited significantly higher cost sensitivity, leading them to minimize cognitive effort by reusing a simplified default policy rather than computing task-specific strategies. This resulted in a systematic bias toward the "easier" Type B tasks and a failure to optimize for Type A tasks.
Clinical Correlates: Lower exerted cognitive effort was associated with higher self-reported anxiety, higher substance use scores, and higher cognitive impulsivity. Lower successor feature learning rates were associated with higher self-rated depression.

Significance and Claims
The paper claims to offer a normative account for atypical learning in addiction that moves beyond the traditional MB–MF dichotomy. The authors argue that addiction does not merely represent a failure of model-based planning or a dominance of model-free habits, but rather a systematic shift in how cognitive resources are allocated.

Specifically, the study posits that:

Successor Features as a Unifying Mechanism: In multi-task environments, human learning (both healthy and pathological) relies on successor features to generalize across tasks, acting as an intermediate strategy between MB and MF.
Resource-Rational Irrationality: The "irrational" behavior observed in SUD (suboptimal policy reuse) is a rational adaptation to a heightened sensitivity to cognitive cost. Individuals with SUD appear to prioritize the conservation of cognitive effort over reward maximization, leading to the rigid reuse of default policies.
Computational Psychiatry Implications: By quantifying cognitive effort via information theory, the study provides a new framework for understanding cognitive control deficits in addiction. It suggests that these deficits may stem not from a lack of cognitive capacity, but from a reduced willingness or ability to expend effort (effort aversion) to update policies away from a default state.

The authors conclude that this resource-rational perspective on successor feature learning offers a more comprehensive explanation for the inflexibility seen in addiction, linking computational mechanisms directly to clinical symptom severity.

More like this