Building Goal-Directed Cognitive Graphs

⚕️

This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Idea: How Your Brain Builds a Map

Imagine you are exploring a new city. You walk around for weeks, noticing every street, alley, and shortcut. Your brain is constantly recording this data: "If I turn left here, I hit a dead end. If I go straight, I see a bakery." This is your dense memory. It's a massive, detailed, and slightly messy collection of every possible path you've ever taken.

But here's the problem: You can't carry a map of every street in your head when you need to get to the bakery right now. You need a simplified, goal-oriented map. You only care about the path to the bakery, not the dead ends or the parks you don't need to visit.

This paper introduces a new theory called the Sparse Cognitive Graph (SCG) to explain how your brain does exactly this. It suggests your brain runs two separate processes at the same time:

The "Notebook" (Gradual Learning): You slowly write down every transition you experience in a dense notebook. This happens gradually and continuously.
The "Highlighter" (Nonlinear Selection): You use a highlighter to pick out only the most important paths from that notebook to create a clean, simple map for decision-making.

The magic of this theory is that small changes in the notebook can lead to sudden, big changes in the map.

The Core Mechanism: The Threshold Switch

Think of the "Notebook" as a bucket filling with water (experience).

Gradual Learning: Every time you take a step, a little bit of water (data) is added to the bucket. This happens slowly.
The Threshold: Imagine there is a line drawn on the bucket. As long as the water is below the line, nothing happens.
The Snap: Once the water crosses that line, a valve opens, and a new "road" appears on your mental map.

Why is this cool?
Because the water level rises slowly, but the moment it crosses the line, the map changes instantly. This explains why humans and animals sometimes seem to learn slowly, and then suddenly, snap, they understand the whole structure and change their behavior completely. It's not a slow shift; it's a sudden reorganization of the map.

The "Reward" Factor: The Dopamine Spotlight

The paper also explains how rewards (like finding money or food) and dopamine (a brain chemical) fit in.

Imagine your "Notebook" is a dark room.

Normal Learning: You walk around, and your flashlight (learning rate) is dim. You slowly see the walls.
Reward: When you find a treasure, your flashlight suddenly turns into a laser beam. You see that specific path very clearly and very quickly.

The theory suggests that dopamine doesn't just tell you "Good job!" (value); it actually speeds up the writing in the notebook for that specific path. This makes it much more likely that the path will cross the "threshold" and get added to your simplified map.

Real-world proof: The researchers tested this on mice. They used light to stimulate dopamine in the mice's brains right after they found a reward. The mice didn't just get "happier"; they instantly changed their strategy, switching to a new path they hadn't used before. The dopamine acted like a turbo-boost for building that specific road on their map.

Solving the "Two-Step" Puzzle

Scientists have long been confused by a famous experiment called the "Two-Step Task."

The Setup: You make a choice, it leads to a second choice, and then you get a reward. Sometimes the first choice leads to the second choice easily (common), and sometimes it's a rare, tricky path.
The Mystery: Humans and mice usually act like they are using two different systems at once: one that just guesses based on luck (Model-Free) and one that plans ahead (Model-Based).
The SCG Solution: This paper says you don't need two systems. You just need one map that changes shape.
- Sometimes the map shows the "common" path as a solid highway.
- Sometimes, after a few tries, the map reorganizes, and the "rare" path becomes a highway too.
- The brain isn't switching between two different brains; it's just updating the map in real-time.

What Does This Look Like in the Brain? (The "Grid" vs. The "Flag")

The paper makes a cool prediction about what neurons look like when they are doing this.

Old Theory (The Grid): If your brain is mapping a city with loops and circles (like a subway system), neurons fire in a grid pattern, like a checkerboard. This is great for general navigation.
New Theory (The Flag): If your brain is building a goal-directed map (like a path to a specific goal), the neurons don't form a grid. Instead, they form "Flags."
- One group of neurons lights up at the Start (the entry point).
- Another group lights up at the Finish (the goal).
- The middle is quiet.

The theory predicts that if you change the map (by adding a new road via dopamine), the "Flags" will move to match the new start and finish points.

Summary: Why This Matters

This paper solves a big puzzle in neuroscience: How do we balance learning everything (stability) with making quick decisions (flexibility)?

The Answer: We keep a detailed, slow-learning "Notebook" of everything we experience. But for making decisions, we only use a "Simplified Map" that is built by highlighting the most important paths.
The Result: This allows us to learn gradually but act decisively. When we finally cross a threshold, our behavior shifts instantly, and our brain reorganizes its internal map to focus entirely on the goal.

It's like having a massive library of books (experience) but only pulling out the specific, highlighted chapters (the graph) you need to solve the problem at hand. And sometimes, a little bit of dopamine is the librarian who suddenly highlights a whole new chapter, changing your entire plan for the day.

1. Problem Statement

Biological intelligence relies on extracting relational structures from experience to guide flexible, goal-directed behavior. While neural evidence suggests that the brain accumulates dense predictive transition statistics (consistent with Successor Representations or SR in hippocampal circuits), behavioral evidence indicates that organisms often rely on sparse, compact internal models for planning and choice.

A central unresolved computational question is: How are gradual, dense transition statistics transformed into a discrete, sparse graph that governs behavior?

The Gap: Standard reinforcement learning (RL) models typically derive valuation and action selection directly from dense predictive maps. However, this fails to explain:
- Abrupt behavioral shifts: Behavior often changes in discrete "regimes" despite gradual learning.
- Multimodal individual differences: Humans exhibit distinct qualitative strategies (e.g., complete reversal vs. no change in revaluation tasks) that cannot be explained by continuous parameter variations alone.
- Cross-species signatures: Standard models struggle to explain specific mouse behavioral patterns in two-step tasks without invoking complex mixtures of model-based and model-free controllers.

2. Methodology: The Sparse Cognitive Graph (SCG)

The authors introduce the Sparse Cognitive Graph (SCG), a reinforcement-learning framework that computationally separates gradual transition learning from nonlinear graph construction.

Core Architecture

The model maintains two parallel representations:

Dense Transition Representation ( $W$ ):
- A dense matrix that accumulates transition statistics gradually via a temporal-difference (TD) rule.
- Unlike the classic SR (which tracks discounted future occupancy), $W$ is anchored to one-step experienced transitions, making it a direct substrate for graph construction.
- Learning rates in $W$ can be modulated by reward (e.g., $\alpha_{\to R}$ vs. $\alpha_{\to NoR}$ ).
Sparse Cognitive Graph ( $G$ ):
- A sparse, directed binary adjacency matrix derived from $W$ via a nonlinear selection rule (thresholding).
- $G_{ij} = 1$ if $W_{ij} \geq \zeta$ (threshold), otherwise $0$.
- Behavioral Control: Valuation and action selection operate exclusively on $G$ , not $W$ . This allows the graph topology to reorganize discretely as transition strengths in $W$ cross the threshold.

Key Mechanisms

Reward-Dependent Bias: The learning rate for transitions preceding a reward ( $\alpha_{\to R}$ ) can differ from those without reward ( $\alpha_{\to NoR}$ ). If $\alpha_{\to R} > \alpha_{\to NoR}$ , reward accelerates the accumulation of predictive strength, biasing the graph construction toward reward-directed paths.
Dopamine as a Modulator: Optogenetic dopamine stimulation is modeled as a transient amplification of the transition learning rate, effectively lowering the threshold for edge formation in $G$ .

3. Key Contributions & Results

A. Explaining Discrete Behavioral Regimes (Human Revaluation Tasks)

Task: Humans learned sequences leading to rewards, then underwent "revaluation" (swapping rewards or transitions) without re-experiencing start states.
Finding: Human data showed bimodal (reward revaluation) and trimodal (transition revaluation) behavioral distributions.
SCG Result: The SCG reproduced these discrete modes using a unimodal distribution of parameters. The nonlinearity of the thresholding step caused smooth changes in learning parameters to trigger abrupt reorganization of graph topology, leading to distinct behavioral strategies (e.g., complete reversal vs. indifference).
Significance: This demonstrates that multimodal behavior can emerge from structural reorganization rather than parameter heterogeneity.

B. Canonical Two-Step Task Signatures (Humans & Mice)

Task: A standard two-step task where agents choose between options leading to common (70%) or rare (30%) transitions, followed by rewards.
Finding: Humans and mice exhibit a "reward-by-transition" interaction (staying after rewarded common transitions, switching after rewarded rare transitions).
SCG Result: The SCG reproduced this interaction without requiring a mixture of model-based and model-free controllers. The interaction emerged naturally from the dynamic reconfiguration of the graph $G$ as transition statistics updated.
Mouse Data Validation: Applied to a mouse dataset with optogenetic dopamine stimulation, the SCG outperformed standard SR, model-free, and model-based RL models.
- Asymmetry: Fitted parameters confirmed $\alpha_{\to R} > \alpha_{\to NoR}$ in mice.
- Dopamine Effect: Stimulating dopamine neurons at the outcome increased the transition learning rate. The model predicted, and data confirmed, that this caused mice to switch more often after rewarded rare transitions (inducing a graph reorganization).

C. Topology-Dependent Population Geometry

Prediction: The SCG predicts that graph topology dictates the geometry of low-dimensional neural population activity.
- Acyclic Graphs: Produce "flag-like" signatures where activity is localized at entry (source) and goal (sink) states.
- Cyclic Graphs: Produce periodic, grid-like signatures (similar to SR predictions in spatial tasks).
Mechanism: These signatures arise from the eigenvectors of the adjacency matrix $G$ . Dopamine-induced changes in graph edges are predicted to shift these population signatures.

4. Significance and Implications

Computational Principle: The paper identifies reward-dependent graph reorganization as a mechanism that reconciles stable, gradual predictive learning with efficient, discrete goal-directed control.
Neural Dissociation: It provides a framework for understanding why neural signatures of learning (in dense $W$ , e.g., hippocampus) may evolve gradually while behavior (governed by sparse $G$ , e.g., prefrontal cortex) changes abruptly.
Role of Dopamine: It offers a novel structural interpretation of dopamine: beyond encoding reward prediction errors (value), dopamine modulates transition learning rates, thereby shaping the topology of the internal cognitive graph.
Efficiency: By restricting planning to a sparse graph, the model achieves computational efficiency (scaling with edges rather than state space size) while retaining the richness of dense predictive learning in the background.
Testable Predictions: The model generates specific, falsifiable predictions regarding low-dimensional population activity (e.g., shifts from grid-like to flag-like structures upon dopamine manipulation) that can be tested with neural recording techniques.

In summary, the SCG framework bridges the gap between dense predictive coding and sparse behavioral control, suggesting that the brain constructs goal-directed behavior by selectively "pruning" a dense predictive map into a functional, sparse graph based on reward history and neuromodulatory signals.