Dual reinforcement-learning network modules for modeling decision-making with multiple strategies

This paper proposes a hybrid deep reinforcement learning (H-DRL) model that unifies model-free and inference-based strategies within a single cortical network, demonstrating how synaptic plasticity and recurrent activity can automatically adapt decision-making behaviors to task demands without requiring an explicit arbitrator.

The Gist

The Big Idea: One Brain, Two Ways of Thinking

Imagine you are trying to navigate a new city. You have two ways to get around:

1. The "Habit" Driver: You just keep turning left because that's what you did yesterday and it worked. You don't think about the map; you just react to the last turn. This is fast and easy, but if the road changes, you get stuck.
2. The "Map" Navigator: You look at the street signs, remember where you've been, and calculate the best route based on how the streets connect. This takes more mental energy but works great when the city layout changes.

Usually, scientists thought our brains used two separate systems for these: one part of the brain for habits and another for planning. But this paper suggests something cooler: Our brain might use just one network that can switch between these two modes automatically, depending on how hard the task is.

The Problem with Old Computer Models

For a long time, computer models of the brain (called Meta-RL) were like the "Map Navigator" only. They were great at learning complex rules and planning ahead. However, real animals (and humans) aren't just perfect planners. We often rely on lazy habits when things are simple, and only switch to deep thinking when things get tricky. The old models couldn't do this mix; they were too rigid.

The Solution: The "Hybrid Deep Reinforcement Learning" (H-DRL)

The authors created a new computer model called H-DRL. Think of this model as a smart robot with a "Dual-Engine" system.

Instead of having two separate brains, this robot has one brain that runs on two different fuel types simultaneously:

1. The "Quick-Change" Engine (Weight-RL): This is like a sticky note on your fridge. Every time you get a reward, you quickly scribble a note: "Do this again!" It's fast, simple, and relies on immediate memory. It's great for repeating patterns.
2. The "Deep-Thinking" Engine (Recurrent-RL): This is like the robot's internal simulation. It keeps a running story of what happened, connecting the dots between past events to predict the future. It's slower but smarter.

The Magic Trick: The robot doesn't need a manager to tell it which engine to use. The task itself decides.

• If the task is simple and repetitive (like a song that plays the same way every time), the robot automatically leans on the Quick-Change Engine. It's "lazy" because it doesn't need to think hard.
• If the task is tricky and changes constantly (like a song that switches genres randomly), the robot automatically switches to the Deep-Thinking Engine to figure out the pattern.

How They Tested It: The Mouse Game

To prove this works, the researchers tested their robot against real mice in a "sound game."

• The Game: Mice heard a sound and had to choose a left or right spout for a treat.
• The Twist: Sometimes the sound pattern repeated (easy), and sometimes it alternated (hard).
• The Result:
  • Real Mice: When the pattern repeated, they just repeated their last choice (Habit). When it alternated, they had to think about the sequence (Planning).
  • Old Robot Model: It tried to plan for everything, even when it was easy. It was inefficient.
  • New H-DRL Robot: It acted exactly like the mice! It used the "lazy" habit engine for the easy part and the "smart" planning engine for the hard part.

The "Silent" vs. "Active" Memory

One of the most fascinating discoveries is how the robot remembers things, which matches what happens in the mouse brain (specifically the Orbitofrontal Cortex, or OFC).

• The "Activity-Silent" Mode (Lazy Learning): When the task is easy, the robot doesn't need to keep its neurons firing constantly to remember the last step. Instead, it changes the strength of the connections between neurons (like tightening a screw). The memory is there, but it's "silent" and doesn't use much energy.
• The "Recurrent-Dynamics" Mode (Rich Learning): When the task is hard, the robot needs to keep a "mental spotlight" on the past. The neurons fire in a specific, active loop to hold the information in working memory.

Why This Matters

This paper changes how we view the brain. It suggests we don't need separate "habit centers" and "planning centers." Instead, we have a single, flexible network that knows exactly when to be lazy and when to be brilliant.

The Analogy Summary:
Imagine a Swiss Army Knife.

• Old theories said you needed a separate hammer for nails and a separate screwdriver for screws.
• This paper says: No, you have one tool that can instantly transform into a hammer when you need to hit a nail, and a screwdriver when you need to turn a screw. The tool itself knows which mode to use based on the job at hand.

This "Hybrid" model helps us understand how animals (and potentially humans) are so good at switching between autopilot and deep focus without getting confused.

Technical Summary

1. Problem Statement

Animals and humans flexibly switch between and integrate multiple behavioral strategies (e.g., model-free vs. model-based, or inference-based) to solve decision-making tasks. However, the neural mechanisms underlying this flexibility remain unclear.

• The Conflict: Traditional neuroscience models often propose distinct brain pathways for different strategies (e.g., striatum for model-free, prefrontal cortex for model-based) or use an explicit "arbitrator" to switch between parallel networks. Conversely, recent deep reinforcement learning (Deep RL) approaches, specifically Meta-RL, suggest that a single recurrent neural network (RNN) can learn optimal strategies.
• The Limitation: Standard Meta-RL frameworks typically enforce a strict separation of timescales: a "first RL" (outer loop) trains the network weights over many episodes, while a "second RL" (inner loop) uses the fixed recurrent dynamics to make decisions during a task. This often results in agents exhibiting purely model-based (inference-based) behavior, failing to capture the mixed strategies (lazy vs. rich learning) observed in biological subjects.

2. Methodology: Hybrid Deep Reinforcement Learning (H-DRL)

The authors propose Hybrid Deep Reinforcement Learning (H-DRL), a modification of the Meta-RL framework that allows a single network to implement dual learning mechanisms without an explicit arbitrator.

Core Architectural Modifications:

1. Removal of Timescale Separation: Unlike standard Meta-RL where weights are updated only between sessions, H-DRL applies trial-by-trial online weight updates.
2. Dual Learning Modules:
   • Weight-RL (Model-Free): Rapid, trial-by-trial synaptic plasticity updates connection weights based on immediate reward prediction errors (RPEs). This acts as a rigid, model-free controller.
   • Recurrent-RL (Model-Based/Inference): The accumulation of these updates over time shapes the RNN's recurrent dynamics, allowing the network to develop flexible, inference-based strategies (rich learning).
3. Implementation Details:
   • Architecture: A Recurrent Neural Network (RNN) using Softplus activation functions to ensure stability.
   • Optimization: Uses simple Stochastic Gradient Descent (SGD) with Truncated Backpropagation Through Time (TBPTT) on a per-trial basis.
   • Task: The model was tested on two tasks:
     • A Two-Step Decision Task (to validate mixed strategy emergence).
     • A Perceptual Decision-Making Task with mice (repeating vs. alternating transition probabilities).

3. Key Contributions

• Unified Framework: Demonstrates that a single cortical network can automatically determine and switch between strategies (model-free vs. inference-based) based solely on task demands, without an external arbitrator.
• Biological Plausibility: Maps the computational components of H-DRL to biological mechanisms:
  • Weight-RL corresponds to activity-silent working memory (synaptic plasticity).
  • Recurrent-RL corresponds to recurrent-dynamics working memory (persistent neural activity).
• Lazy vs. Rich Learning: The model naturally exhibits "lazy learning" (adjusting only output weights) for simple, repetitive tasks and "rich learning" (updating internal recurrent dynamics) for complex, changing tasks.

4. Key Results

A. Two-Step Decision Task Validation

• H-DRL successfully reproduced the mixed strategies observed in humans and animals (a combination of model-free and model-based choices).
• In contrast, standard Meta-RL models tended to converge on purely model-based strategies.
• The balance between strategies in H-DRL emerged automatically from the task structure, not from tuned hyperparameters.

B. Perceptual Decision-Making Task (Mouse Data)
The model was tested against behavioral data from mice performing a tone discrimination task under two conditions:

1. Repeating Condition ($p=0.2$): The tone category repeats frequently.
   • Behavior: Mice used a simple model-free strategy (repeat the last rewarded choice).
   • H-DRL: Adopted lazy learning. It relied heavily on Weight-RL (updating output weights) and showed minimal changes in recurrent dynamics.
2. Alternating Condition ($p=0.9$): The tone category switches frequently.
   • Behavior: Mice required an inference-based strategy to track state transitions.
   • H-DRL: Adopted rich learning. It significantly updated recurrent connections (Recurrent-RL) to encode state transitions.

C. Perturbation Analysis

• Weight-Freeze Test: Freezing weights (disabling Weight-RL) disrupted performance in the repeating condition but not the alternating condition.
• Activity-Reset Test: Resetting hidden states (disabling Recurrent-RL) disrupted performance in the alternating condition but not the repeating condition.
• Conclusion: The model automatically selects the appropriate learning module (Weight-RL vs. Recurrent-RL) based on task complexity.

D. Neural Correlates (OFC Activity)
The authors compared H-DRL unit activity with electrophysiological recordings from the mouse Orbitofrontal Cortex (OFC):

• Repeating Condition: Mouse OFC neurons showed activity-silent behavior. Decoding of previous choices dropped to chance levels during the inter-trial interval (ITI), suggesting memory was stored in synaptic weights (consistent with Weight-RL).
• Alternating Condition: Mouse OFC neurons maintained persistent activity during the ITI, allowing successful decoding of previous events (consistent with Recurrent-RL).
• Model Alignment: H-DRL unit activity mirrored these condition-dependent modes, whereas standard Meta-RL failed to replicate the condition-specific decay of activity.

5. Significance

• Theoretical Unification: H-DRL bridges the gap between "dual-system" theories (distinct brain regions for different strategies) and "single-network" theories (one network doing everything). It suggests that the same network can implement both systems via different learning timescales (synaptic plasticity vs. recurrent dynamics).
• Neurobiological Insight: The study provides a computational explanation for the coexistence of "activity-silent" and "activity-based" working memory in the prefrontal cortex (specifically OFC). It suggests that the brain dynamically switches between these modes depending on whether a task requires simple habit formation or complex inference.
• Methodological Advance: It offers a more biologically realistic Deep RL framework that captures the nuance of animal behavior better than standard Meta-RL, which often over-optimizes for purely model-based solutions.

In summary, this paper proposes that the brain does not need separate circuits or an explicit switch to handle multiple strategies; rather, a single recurrent network can automatically toggle between "lazy" (synaptic) and "rich" (dynamic) learning modes to optimize performance across varying task demands.