Adaptive integration of model-based and model-free strategies in human reinforcement learning of reachable space

Using a novel robotic maze task, this study demonstrates that humans adaptively integrate model-based and model-free reinforcement learning strategies to navigate reachable space, shifting toward more efficient model-free control as familiarity increases and revealing that spatial learning architectures are shared across scales but calibrated to the specific constraints of the effector system.

The Gist

The Big Idea: How We Learn to Move Our Hands

Imagine you are trying to get a coffee cup from a crowded table. You have to reach out, dodge a stack of plates, and avoid knocking over a vase. This happens every day, but scientists haven't really studied how our brains learn to do it until now.

This paper asks a simple question: When we learn to navigate a tricky space with our hands, does our brain use a "GPS map" (planning ahead) or does it just rely on "muscle memory" (trial and error)?

The answer is: We use both, and we switch between them like a hybrid car.

---

The Experiment: The Robot Maze

To figure this out, the researchers built a special video game.

• The Setup: Participants sat in front of a robotic arm. They held a handle that controlled a virtual ball on a screen.
• The Goal: Move the ball from a starting point to a red target square.
• The Obstacle: The maze was full of grey blocks. If you hit a block, the robot would push back against your hand (like hitting a wall).
• The Twist: They tested two groups:
  1. The "Eyes-On" Group: They could see the whole maze and the blocks.
  2. The "Blind" Group: They couldn't see the maze or their hand. They had to feel their way around using only the robot's push-back feedback.

The Two Brain Modes: The Architect vs. The Habit-Former

The researchers used computer models to guess what was happening in the participants' brains. They found two distinct strategies:

1. Model-Based (The Architect):

   • What it is: This is your brain building a mental map. It's like a GPS. You look at the maze, plan a route, and think, "If I go left, then up, I'll hit the wall, so I'll go right instead."
   • Pros: It's flexible. If the maze changes, you can instantly recalculate.
   • Cons: It's slow and tiring. It takes a lot of mental energy to plan every step.

2. Model-Free (The Habit-Former):

   • What it is: This is your brain caching successful moves. It's like a squirrel remembering where it buried nuts. "I went left here before, and I got the reward, so I'll go left again." It doesn't know why it works, it just knows that it works.
   • Pros: It's super fast and automatic.
   • Cons: It's rigid. If the maze changes, the squirrel keeps digging in the wrong spot.

The Surprising Findings

1. We Start as Architects, Then Become Habit-Formers

At the very beginning of the game, everyone used the Architect strategy. They were planning carefully. But as they played more and more rounds, they slowly switched to the Habit-Former.

• The Analogy: Think of learning to drive a new route. At first, you stare at the GPS (planning). After a week, you drive on autopilot without thinking (habit). The study shows our brains do this automatically to save energy.

2. The "Blind" Group Relied More on Habits

The group that couldn't see the maze relied on the Habit-Former strategy much more than the group that could see.

• Why? When you can't see, building a perfect mental map is hard and uncertain. So, your brain says, "I'll just trust my muscle memory for this one."
• The Twist: Even the group that could see the maze eventually switched to habits! This proves that our brains switch to habits not just because we are confused, but because planning is expensive. Once we know the way, we stop planning to save mental energy.

3. Speed vs. Safety

Here is the coolest part: The people who relied more on Habits moved faster.

• The Analogy: The "Architect" is like a chess player thinking for 10 minutes before moving a piece. The "Habit-Former" is like a reflex.
• The study found that when people moved faster, they were actually using their "habit" brain more. Interestingly, these fast movers also bumped into fewer blocks. Why? Because habits repeat what worked before, avoiding the risky, unexplored paths that the "Architect" might try to calculate.

The Big Comparison: Hands vs. Feet

The researchers compared their hand-maze game to a similar game where people "walked" through a virtual maze using a chair.

• The Result: People relied much more on habits when using their hands than when using their feet.
• The Reason: Walking is slow and tiring. If you take a wrong turn while walking, it costs you a lot of time. So, your brain forces you to plan carefully (Architect mode).
• The Hand Difference: Moving your hand is fast and cheap. If you take a slightly wrong turn with your hand, it only costs a split second. Because the "penalty" for a mistake is so low, your brain feels safe enough to just use fast, automatic habits.

The Takeaway

Our brains are incredibly smart engineers. They don't just stick to one way of learning.

• When we are learning something new, we plan (Architect).
• Once we get the hang of it, we switch to habits (Habit-Former) to save energy and move faster.
• We switch to habits even faster when the task is quick and easy (like moving a hand) compared to when it's slow and costly (like walking).

In short: We are all hybrid learners. We build maps when we need to, but as soon as we can, we let our habits take the wheel so we can move faster and free up our brains for other things.

Technical Summary

1. Problem Statement

Most skilled human behavior occurs within reachable space (the area immediately surrounding the body where hands interact with objects). While reinforcement learning (RL) frameworks have extensively studied spatial learning in large-scale navigation (distinguishing between Model-Based (MB) planning and Model-Free (MF) habit formation), the computational mechanisms governing learning in reachable space remain unexplored.

Key questions addressed:

• How do humans integrate MB and MF strategies when learning to navigate obstacles in reachable space?
• Does the balance between these strategies shift dynamically during learning (e.g., from MB to MF)?
• How does the effector system (hand vs. locomotion) and sensory modality (vision vs. haptics) influence the arbitration between MB and MF control?

2. Methodology

Experimental Paradigm

The authors utilized a novel robotic maze task involving a 3D robotic manipulandum (3BOT) and a Virtual Reality (VR) display.

• Task: Participants moved a virtual sphere (controlled by a handle) from a start point to a target within a $10 \times 10$ grid maze, avoiding blocks.
• Conditions:
  1. Visual-Haptic: Participants could see the maze layout and feel haptic feedback upon collision.
  2. Haptic: The maze blocks and hand position were invisible; participants learned the layout solely through haptic feedback (touch).
• Procedure: 18 participants per condition performed 10 trials across 25 unique mazes.

Computational Modeling

The authors fitted several RL algorithms to the discretized movement trajectories:

1. Model-Based (MB): Uses a transition function $T(s,a)$ learned from experience (or vision in the Visual-Haptic condition) to perform offline value iteration (planning).
2. Model-Free (MF): Uses Q-learning with eligibility traces to cache action values without an internal model of the environment.
3. Hybrid Models: To capture the mixture of strategies, three hybrid models were tested:
   • Hybrid-Constant (HC): A fixed weight combining MB and MF policies.
   • Hybrid-Dynamic (HD): The MF weight is a logistic function of the trial number (allowing a shift over time).
   • Hybrid-Stepwise (HS): An independent MF weight for every action step (non-parametric, capturing fine-grained dynamics).

Comparative Analysis

The modeling approach was applied to a previously published Virtual Reality Navigation task (de Cothi et al., 2020) with identical maze configurations but involving locomotion (swivel chair) and limited field-of-view vision, allowing for a direct cross-domain comparison.

3. Key Results

Strategy Shifts and Dynamics

• MB to MF Transition: Across both conditions, participants dynamically shifted from MB to MF strategies as learning progressed. The Hybrid-Dynamic (HD) and Hybrid-Stepwise (HS) models provided the best fit (lowest BIC), outperforming single-strategy models.
• Sensory Influence: The shift to MF was more pronounced in the Haptic condition (no vision) than in the Visual-Haptic condition. This suggests that uncertainty in the environmental model drives reliance on MF strategies.
• Step-Level Modulation: MF reliance was not static; it increased with:
  • State Familiarity: More visits to a specific grid location increased MF weight.
  • Distance to Goal: States farther from the target showed higher MF reliance (likely due to increased planning complexity/uncertainty in MB).

Behavioral Correlates

• Speed vs. Optimality: Participants with higher MF reliance moved faster but took less optimal paths (in the Visual-Haptic condition). This supports the interpretation that MF control reduces computational load, enabling faster execution at the cost of global optimality.
• Collision Reduction: Higher MF reliance correlated with fewer obstacle contacts. MF strategies tend to repeat successful sequences (conservative trajectories), whereas MB strategies might explore unverified paths based on imperfect internal models, leading to collisions.
• Simulation Performance: Pure MF algorithms failed to solve the maze independently (performing near random). However, in hybrid models, MF components effectively "imitated" successful paths initially generated by MB planning, suggesting a cooperative scaffolding mechanism.

Cross-Domain Comparison (Reachable vs. Navigable Space)

• Stronger MF in Reachable Space: Despite identical maze structures, participants relied significantly more on MF strategies in reachable space than in the navigation task.
• Distance Effect: In reachable space, MF weight increased with distance to the goal; in navigation, it did not.
• Interpretation: Hand movements are biomechanically cheaper and faster than locomotion. The marginal cost of suboptimal paths is lower for hands, reducing the incentive for computationally expensive MB planning compared to navigation.

Alternative Models

• Successor Representation (SR): SR models (occupying the spectrum between MB and MF) were tested but performed worse than the hybrid models, indicating that a weighted mixture of distinct MB and MF systems better explains the data than a single predictive map.

4. Key Contributions

1. Novel Domain Application: First demonstration of adaptive MB/MF integration specifically in reachable space, bridging the gap between motor control and spatial cognition.
2. Dynamic Arbitration Framework: Developed and validated hybrid models (specifically HD and HS) that capture step-by-step strategy shifts, moving beyond the assumption of fixed strategy weights.
3. Effector-Specific Calibration: Established that the MB/MF arbitration is not a fixed property of the learner but is calibrated to the effector system. The lower cost of hand movements favors MF strategies more than the high-cost locomotion in navigation.
4. Behavioral Signatures: Linked computational parameters to specific behavioral metrics (speed, collision rates), providing empirical evidence that MF control facilitates faster, more conservative motor execution.

5. Significance

• Theoretical Impact: The findings challenge the view that MB/MF arbitration is solely driven by environmental uncertainty. Instead, they propose that computational costs and biomechanical constraints of the effector system are primary drivers.
• Motor Learning: The study formalizes the qualitative shift from "slow, deliberative control" to "fast, automatic performance" observed in motor learning within a rigorous RL framework.
• Clinical Relevance: The paradigm offers a new tool for investigating disorders affecting motor learning and RL (e.g., Parkinson's disease, OCD, stroke), particularly those involving basal ganglia or hippocampal circuits, by testing how these conditions affect learning in the space most critical for daily life (reachable space).
• Neural Implications: The results suggest that while navigation relies heavily on hippocampal cognitive maps, reachable space learning may rely more on parietal-premotor circuits, though the specific neural substrates for MB/MF arbitration in this domain remain an open question for future neuroimaging.