Efficient Learning of Predictive Maps for Flexible Planning

This paper introduces the Successor Representation with Importance Sampling (SR-IS), a novel model that constructs policy-independent predictive maps to enable efficient, flexible planning and better explains human replanning biases compared to existing theories.

The Gist

Imagine your brain is like a highly skilled tour guide trying to navigate a massive, ever-changing city. To get from point A to point B quickly, the guide needs a mental map.

The Old Problem: The "Routine" Map
For a long time, scientists thought the brain used a specific type of map called the "Successor Representation." Think of this like a commuter's subway map. It's incredibly useful if you always take the same route to work every day. It tells you exactly which stations you'll pass through based on your usual habits.

But here's the catch: this map is rigid. If the city suddenly closes a tunnel or you decide you want to go to a different neighborhood, the commuter map is useless. It's so tied to your old routine that it can't easily help you plan a new trip. In the paper's terms, this map is "policy-dependent," meaning it only works if you stick to your current plan.

The New Solution: The "Explorer's" Map (SR-IS)
The authors of this paper introduce a new model called SR-IS (Successor Representation with Importance Sampling). You can think of this as upgrading from a rigid subway map to a dynamic, GPS-style explorer's map.

Instead of just recording where you usually go, this new model uses a clever trick (called "importance sampling") to learn the layout of the entire city, regardless of which path you are currently walking. It's like a guide who learns the streets by observing traffic patterns and potential detours, not just by following their own daily commute.

Why This Matters
Because this new map isn't stuck on one specific route, it has two superpowers:

1. Instant Adaptation: If the environment changes (like a road closure), the guide can instantly update the map and suggest a new path without having to relearn the whole city from scratch.
2. Better Human Simulation: The paper shows that this model explains how humans actually think better than the old models. When people are asked to change their plans, they don't just flip a switch; they show "graded biases" (small, logical hesitations or adjustments). The old maps couldn't explain these subtle human quirks, but the new SR-IS model matches human behavior perfectly.

The Bottom Line
This paper bridges the gap between how we think the brain builds mental maps and how we actually behave when we need to be flexible. It suggests that our brains might be using a smart, adaptable system (SR-IS) that allows us to learn the structure of our world once, and then use that knowledge to plan any future journey, no matter how the rules change.

Technical Summary

Technical Summary: Efficient Learning of Predictive Maps for Flexible Planning

Problem Statement
Cognitive maps are essential for flexible behavior, as they provide reusable internal representations of task structure. The successor representation (SR) has been proposed as a mechanism for computing these maps by encoding expected future state occupancy. However, a critical limitation of the standard SR is its policy dependence; because it is learned under a specific decision policy, it fails to support truly flexible planning when that policy changes or when the environment is altered. This constraint hinders the ability to rapidly adapt behavior to new goals or environmental dynamics.

Methodology
To address the limitations of policy-dependent representations, this paper introduces a new model: the Successor Representation with Importance Sampling (SR-IS). This approach integrates temporal-difference (TD) learning with importance sampling techniques. The core mechanism allows the model to construct predictive maps that are independent of the agent's current decision policy. By decoupling the learning of environmental structure from the specific policy being executed, SR-IS can learn the underlying topology of the environment even while the agent is exploring or acting sub-optimally. These representations are designed to be efficiently updated in response to environmental changes, facilitating rapid behavioral adaptation.

Key Contributions

• Novel Algorithm: The proposal of SR-IS, which combines TD learning and importance sampling to generate policy-independent predictive maps.
• Theoretical Integration: Bridging the gap between predictive map theories and observed planning behaviors, specifically addressing the rigidity of previous SR models.
• Human Behavior Modeling: Providing a computational account for "graded biases" observed in human replanning, a phenomenon that existing models failed to explain.

Results
The authors demonstrate that SR-IS outperforms existing models in planning tasks. Specifically, the model successfully accounts for the graded biases found in human replanning data, offering a more accurate fit to empirical observations than prior approaches. The results indicate that SR-IS enables more efficient updates when the environment changes, leading to faster adaptation compared to policy-dependent alternatives.

Significance
The paper claims that this work offers new insights into flexible decision-making in the brain. By resolving the issue of policy dependence in predictive maps, SR-IS provides a more plausible mechanism for how the brain might compute reusable representations of task structure. This advancement suggests a pathway for understanding how agents can maintain flexible behavior and rapidly adapt to changing conditions without being constrained by their immediate decision policies.