🧬 biology

Cortex and subcortex play distinct roles over learning when cortical memory is limited

This paper proposes a theoretical framework demonstrating that when cortical memory is limited, optimal learning emerges from a functional dissociation where the cortex captures the environment's general structure while subcortical circuits specialize in reward-based learning, a hypothesis that can be tested with experimental data.

Original authors: Matthew Farrell, Taro Toyoizumi

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Matthew Farrell, Taro Toyoizumi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine your brain is a high-tech office with two distinct departments working together to help you make decisions: the Cortex (the "Executive Office") and the Subcortex (the "Fast-Action Crew").

This paper explores a simple but powerful idea: The Executive Office is incredibly smart but has very limited desk space (memory), while the Fast-Action Crew is less flexible but has endless storage.

Here is how the paper breaks down their relationship using a simple game.

The Game: A Tree of Choices

The researchers set up a mental game that looks like a tree.

You start at the trunk (the root).
You have to make a series of choices to move down the branches.
At the very end of the branches (the leaves), there might be a reward (like a cookie).
The problem? The tree is huge, and the "Executive Office" (the Cortex) can only remember a few specific paths at a time. It can't memorize every single branch.

The Two Departments

The Fast-Action Crew (Model-Free/Subcortex): This team learns by trial and error. "If I go left, I get a cookie. If I go right, I get nothing." They are great at repeating what works, but they are slow to learn if the rules change. They need to eat the cookie many times to be sure it's there.
The Executive Office (Model-Based/Cortex): This team tries to build a map of the whole tree. They don't just wait for cookies; they try to understand how the tree works. "If I go left, there's a 70% chance I'll hit a dead end, but a 30% chance I'll find a path to the cookie." This is very powerful, but because their "desk space" (memory) is limited, they can't draw the whole map. They have to choose which parts of the tree to draw.

The Big Discovery: How to Use Limited Desk Space

The paper asks: If the Executive Office can only remember a few branches, which ones should it draw?

The researchers tested two different strategies for filling that limited desk space:

Strategy A: "The Cookie Hunter" (MAXREWARD)

The Executive Office only draws the branches that lead directly to the cookie right now.

Pros: It's very efficient at getting the current reward.
Cons: If the cookie suddenly moves to a different part of the tree, this team is stuck. They have to erase their old map and start drawing a new one from scratch. They are slow to adapt.

Strategy B: "The Explorer" (MAXREACH)

The Executive Office ignores where the cookie is for a moment. Instead, it draws the branches closest to the trunk (the start of the tree).

Pros: It builds a general understanding of the "structure" of the tree. It knows how to get from the start to any part of the tree.
Cons: It might waste time drawing paths that don't currently have cookies.
The Magic: When the cookie moves to a new spot, the Explorer team is ready. Because they already drew the main paths near the start, they can instantly figure out the new route to the cookie without starting over.

The Verdict: When to Use Which?

The paper found that Strategy B (The Explorer) is often better in a changing world.

If the environment is stable (the cookie stays in the same spot), the "Cookie Hunter" wins. But if the cookie moves around often (which happens a lot in real life), the "Explorer" wins because it focuses on learning the general structure of the environment rather than just the specific reward.

What This Means for the Brain

The authors suggest this explains why our brain is built the way it is:

The Cortex (Executive Office) is specialized for learning the general structure of the world. It builds the map. It doesn't need a cookie on every branch to keep drawing the map; it just needs to know how the tree grows.
The Subcortex (Fast-Action Crew) is specialized for reward-based learning. It focuses on the specific paths that lead to the cookie.

This separation allows the brain to be efficient. The Cortex doesn't waste its limited memory trying to memorize every single reward location. Instead, it builds a flexible map, while the Subcortex handles the specific "where is the treat?" details.

How to Test This

The paper also suggests a way to test this in real people. If you watch someone playing this tree game:

If they are a "Cookie Hunter," they will struggle immediately when the reward moves.
If they are an "Explorer," they will adapt quickly to the new reward location because they were already paying attention to the main paths near the start of the game.

In short: To survive in a changing world, it's better to learn the map of the territory than to just memorize where the treasure is today.

Technical Summary: Cortical and Subcortical Roles in Learning Under Memory Constraints

Problem Statement
The paper addresses the computational trade-off between high-power, resource-intensive cortical processing (associated with flexible, model-based reasoning) and lower-cost, automatic subcortical processes (associated with model-free learning). While it is hypothesized that the brain integrates these systems for resource efficiency, existing theoretical frameworks often fail to explicitly constrain the memory resources of the model-based (MB) module. In many complex environments, the number of state-transition probabilities exceeds the memory capacity of higher-order thinking functions. The authors investigate how a MB learner, explicitly constrained by limited memory capacity, should allocate its resources when learning alongside a model-free (MF) learner. Specifically, they ask whether the MB system should prioritize tracking transitions directly associated with current rewards or broader environmental structures.

Methodology
The authors propose a theoretical framework using a tree-structured environment with nondeterministic transitions.

Environment: A regular tree with depth $d$ , where an agent starts at a root state $s_0$ and moves to leaf states. Rewards are only assigned at leaf states. The environment is divided into two phases: Phase 1 (fixed reward pattern) and Phase 2 (shifted reward pattern).
Learners:
- Model-Free (MF): Uses a SARSA(0)-style rule with a backward-propagating update, allowing immediate reward propagation through the tree without extra parameters like $\lambda$ .
- Model-Based (MB): Uses dynamic programming to compute Q-values based on estimated transition probabilities $\hat{P}$ . Crucially, the MB learner is constrained by a memory capacity $m$ , limiting the number of edges (transitions) it can track and estimate.
Memory Allocation Strategies: The study evaluates two distinct strategies for selecting which edges to track within the memory limit $m$ $m$ :
1. MAXREWARD: Prioritizes edges associated with reward (those on trajectories leading to positive value), using proximity to the root as a tie-breaker.
2. MAXREACH: Prioritizes edges closest to the root state ( $s_0$ ), regardless of immediate reward association, to maximize the ability to navigate to arbitrary goal states.
Evaluation: Performance is measured by the average reward received over episodes. The study also introduces a "consistency" metric to compare hypothesized strategies against generated data, allowing for fine-grained analysis of action choices at specific tree levels.

Key Results

Trade-off in Static vs. Dynamic Environments:
- In Phase 1 (stable reward), MAXREWARD generally outperforms or equals MAXREACH because it focuses resources on exploiting the current reward structure.
- In Phase 2 (after reward location shifts), MAXREACH demonstrates a significant advantage. Because it tracks the general structure near the root, it retains knowledge of transition probabilities that become relevant when the reward moves. MAXREWARD, having discarded non-rewarded edges in Phase 1, must relearn these transitions, resulting in a temporary performance drop.
Memory Capacity Effects: When memory capacity $m$ is sufficiently high, both strategies converge in performance. However, under strict constraints, the choice of strategy dictates the agent's adaptability to non-stationary environments.
Consistency Signatures: The analysis of strategy consistency reveals a distinct temporal signature. When comparing a MAXREACH agent's data against a MAXREWARD hypothesis (and vice versa), the consistency of the first-stage action (root level) drops dramatically immediately after the reward shift in Phase 2, before gradually recovering. This provides a specific, testable signature for distinguishing between the two learning modes.

Significance and Claims
The paper claims to provide a theoretical foundation for a functional dissociation between cortical and subcortical systems during learning:

Functional Specialization: The authors propose that the cortex (MB) supports the learning of general environmental structure (MAXREACH), while subcortical circuits (MF) specialize in reward-based learning.
Neurobiological Plausibility: This framework offers an explanation for why cortical areas receive sparser dopamine projections compared to the striatum. If the cortex focuses on structural learning independent of immediate reward, it requires less direct reward signaling than the MF system, which relies on repeated reward experiences to update values.
Experimental Testability: The authors detail a method to test these hypotheses using experimental data. By measuring the consistency of an animal's actions with MAXREWARD or MAXREACH strategies, researchers can identify whether the subject is prioritizing reward exploitation or structural learning, particularly observing the specific "dip" in consistency at the root level following environmental changes.
AI Implications: The work suggests that efficient AI systems might benefit from similar resource-constrained architectures, where a high-cost planning module focuses on general structure while a low-cost module handles immediate reward optimization.

The authors remain modest, acknowledging that their current model uses a toy environment and tabular methods. They note that future work should explore parametric models, more complex signal integration, and the mechanisms for handing over control from MB to MF as tasks are mastered.