Learning to select computations in recurrent neural circuits

This paper presents a recurrent neural network model that learns to select computations by integrating rational meta-reasoning with meta-learning, thereby offering a mechanistic account of how the brain achieves flexible and efficient adaptive control of thought.

The Gist

Imagine your brain is a busy, high-stakes kitchen. You have a limited amount of time, energy, and ingredients (cognitive resources) to cook a meal (make a decision). Sometimes, you need to decide quickly: "Do I buy this popcorn?" Other times, you need to plan a complex route: "If I go to the store, then the bank, then the park, will I make it on time?"

The big question scientists have asked for years is: How does the brain know which mental tools to use, and when to use them, without burning out?

This paper introduces a new "smart kitchen robot" (a computer model) that learns to answer this question. Here is the story of how it works, explained simply.

1. The Problem: The "Thinking" Tax

Think of your brain as a CEO. The CEO has two types of employees:

• The Doers: These are your physical actions (buying the popcorn, walking to the store).
• The Researchers: These are your mental actions (recalling a memory, simulating a future, checking a map).

The problem is that "Researching" costs time and energy. If you spend 10 minutes mentally simulating every possible path to the park, you might be too tired to actually walk there. If you don't think enough, you might get lost.

The brain needs a way to decide: Should I think more? Should I stop thinking and just act? And exactly what should I think about?

2. The Solution: A Robot That Learns to "Think"

The authors built a computer brain (a Recurrent Neural Network) and taught it a special trick called Meta-Learning.

Usually, robots learn to do a task (like playing chess). This robot learned how to learn. It was given a special rule: "You can pay a small 'tax' (cost) to ask your internal database for information. But if you pay too much tax, you lose the game."

• The "Information Generator": Imagine the robot has a magical librarian. When the robot asks, "What was the price of popcorn last time?", the librarian instantly pulls up the memory. But asking the librarian takes a tiny bit of the robot's battery.
• The Goal: The robot had to learn to ask the librarian just enough questions to make a good decision, but not so many that it ran out of battery.

3. The Experiments: From Popcorn to Planning

The researchers tested this robot in two very different scenarios to see if it acted like a human.

Scenario A: The Popcorn Dilemma (Simple Choice)

The Task: The robot had to choose between two snacks. It didn't know their true value, so it had to "glance" at them to get a noisy guess (like a blurry photo).
The Human Habit: Humans tend to look at the snack they are least sure about, or the one that is closest in value to the other. We don't stare at the one we already know is terrible.
The Robot's Success: The robot learned this exact strategy! It stopped wasting time looking at the "bad" snack and focused its mental energy on the "uncertain" ones.
The Brain Connection: When the researchers looked at the robot's internal "thought patterns," they looked surprisingly similar to the electrical signals recorded in the orbitofrontal cortex (a part of the monkey brain responsible for decision-making). The robot's "thinking" looked just like a monkey's brain thinking.

Scenario B: The Treasure Hunt (Complex Planning)

The Task: The robot had to navigate a maze of choices to find the most treasure. It could only see the next step if it "looked" at it.
The Human Habit: Humans don't check every single path in the maze (that takes too long). We use a "best-first" strategy: we look at the path that looks most promising right now, and if it looks good, we go deeper. We also tend to look at things close to where we are standing.
The Robot's Success: The robot learned to do the same thing. It didn't check every dead end. It focused on the most promising paths.
The Brain Connection: In a real human study, scientists found that when people plan, their hippocampus (memory center) and prefrontal cortex (planning center) work together in a specific rhythm, simulating steps one by one. The robot, when it "thought" through the maze, showed the exact same rhythmic pattern in its internal code.

4. The Big Idea: Thinking is Just Learning from Yourself

The most exciting part of this paper is a new way of looking at "thinking."

Usually, we think of Learning as getting information from the outside world (like studying a textbook).
But this paper suggests that Reasoning is just Learning from your own thoughts.

• The Analogy: Imagine you are trying to solve a puzzle. Instead of asking a friend for help, you ask yourself, "What if I move this piece here?" You get an answer from your own mind.
• The robot learned that every time it "thought" (queried its memory), it was essentially collecting a new data point to help it learn the rules of the game.

Why Does This Matter?

This paper bridges a huge gap between two worlds:

1. Mathematical Theory: The idea that humans are "rational" and try to save energy.
2. Biological Reality: The messy, electrical firing of neurons in our brains.

It shows that you don't need a magical "little man" inside your head to tell you what to think. Instead, your brain is a learning machine that has figured out how to learn how to learn. It treats its own thoughts as experiments, gathers data from them, and gets better at deciding what to think next.

In short: Your brain isn't just a calculator; it's a scientist that runs experiments on its own ideas to figure out the best way to solve problems, all while trying to save energy. This robot proved that a simple system can learn to do exactly that.

Technical Summary

1. Problem Statement

Biological intelligence is characterized by flexibility and efficiency, achieved through meta-reasoning: the ability to adaptively decide what to think and when to think it, balancing external utility against computational cost. Despite progress in resource-rational analysis, two major gaps remain in understanding how the brain implements this:

1. Algorithmic Challenge: Identifying optimal computational strategies is itself computationally expensive. Previous approaches often rely on hand-engineered strategies or ignore neural plausibility.
2. Representational Challenge: Most meta-reasoning models assume symbolic, hand-specified representational spaces (e.g., Bayesian posteriors). It remains unclear how meta-reasoning operates in distributed neural systems where representations are learned and dynamic.

The core question is: How can a neural system learn to select and execute internal computations (mental actions) to solve complex problems efficiently, without relying on pre-specified symbolic architectures?

2. Methodology

The authors propose a framework that unifies Rational Meta-Reasoning with Meta-Reinforcement Learning (Meta-RL).

Core Architecture

• Model: A Recurrent Neural Network (RNN) using Gated Recurrent Units (GRUs) with an Actor-Critic architecture.
• Action Space: The agent's actions are divided into two types:
  • Physical Actions: Interact with the external environment (e.g., choosing an item).
  • Mental Actions (Computations): Leave the environment unchanged but query an "Information Generator."
• Information Generator: A task-specific module (abstracting brain regions like the hippocampus or basal ganglia) that returns decision-relevant information (e.g., recalling a memory, simulating a future state) to the RNN's input at the next time step.
• Objective: The agent is trained via policy gradient (REINFORCE with baseline) to maximize expected cumulative reward, which includes:
  • External Utility: Rewards from physical actions.
  • Computational Cost: Negative rewards (penalties) incurred for every mental action (simulating time/effort).
• Learning Mechanism: The RNN parameters are learned slowly across tasks (meta-learning), but during evaluation, the agent adapts rapidly to new problems solely through its recurrent dynamics, without updating weights.

Experimental Tasks

The framework was tested across three distinct domains:

1. Simple Choice Task (Binary/Trinary): Based on human eye-tracking data (Callaway et al., 2021). Agents must sample noisy value estimates of items before choosing.
2. Macaque Orbitofrontal Cortex (OFC) Task: Based on neural recordings (Rich & Wallis, 2016; McGinty & Lupkin, 2023). Agents simulate the deliberation process of macaques choosing between options.
3. Multi-Step Planning Task: Based on a tree-search graph navigation task (Callaway et al., 2024b) and a mental simulation task (Vikbladh et al., 2024). Agents must plan paths or simulate sequences to maximize rewards.

3. Key Contributions

1. Unification of Frameworks: The paper bridges meta-reasoning (normative theory of computation selection) and meta-learning (learning to learn). It posits that reasoning is a form of learning from information generated by one's own cognitive operations.
2. Neural Implementation of Meta-Reasoning: It provides a mechanistic account of how adaptive control can be implemented in neural systems. The RNN learns to represent belief states and update them dynamically through recurrent dynamics, rather than relying on hand-coded Bayesian updates.
3. Mental Actions as Internal Experiments: The model formalizes computations as "internal actions" that generate information to reduce uncertainty, analogous to physical actions in the environment.
4. Biological Plausibility: The architecture maps the RNN to the Prefrontal Cortex (PFC) and the Information Generator to other regions (Hippocampus, Basal Ganglia), offering a testable hypothesis for PFC-Hippocampal interactions in planning.

4. Key Results

A. Simple Choice Task (Behavior & Belief States)

• Strategic Sampling: The agent learned to sample items with high uncertainty or those close in value to the current best option, matching the optimal symbolic model and human eye-tracking data.
• Belief State Representation: Linear decoders could perfectly extract Bayesian sufficient statistics (posterior means and precisions) from the RNN's hidden states.
• Geometric Structure: Principal Component Analysis (PCA) revealed that the hidden states organized into a grid-like structure where dimensions corresponded to posterior means of items, and a third dimension encoded the currently attended item. This demonstrated that the network learned to represent and update belief states geometrically.

B. Macaque OFC Dynamics

• Alternating Value States: The agent reproduced the alternating pattern of value representations observed in macaque OFC during deliberation (switching between chosen and unchosen option values).
• Statistical Signatures: The agent matched macaque data in the number of value states, the relationship between value magnitude and state transitions, and error rates.
• Temporal Subspaces: The model captured the sequential emergence of value gradients and the rotation of value gradients over time, supporting the theory that PFC uses "structured slots" or temporal subspaces to represent sequential information.

C. Planning Task (Human-like Strategies)

• Tree Search Behavior: In the graph navigation task, the agent adopted a "best-first" search strategy, preferentially querying states with high path values and shallow depths, mirroring human behavior.
• Local Value Backups: Analysis of decision logits showed the agent performed local value updates (Bellman-like backups) upon querying a state. Notably, these backups were optimistic, propagating positive updates to parents but ignoring negative ones, a strategy that improved efficiency.
• Mental Simulation (Rollouts): In the revaluation task, the agent learned to perform "rollouts" (sequentially simulating future states).
• Neural Dynamics Match: The agent's hidden state dynamics exhibited the same lagged correlation structure as human MEG data. When simulating a sequence starting at state $N$, the dynamics matched those of a sequence starting at $N-1$ but shifted by one time step, confirming the agent was performing step-by-step mental simulation similar to humans.

5. Significance

• Mechanistic Explanation: The work moves beyond abstract theories of meta-reasoning to provide a concrete, neurally plausible mechanism (recurrent dynamics + meta-RL) for how the brain selects computations.
• Bridging Scales: It successfully links high-level cognitive theories (resource rationality) with low-level neural phenomena (OFC value alternation, PFC temporal subspaces, and hippocampal rollouts).
• Efficiency in AI: By demonstrating that flexible, efficient computation can emerge in a brain-like recurrent architecture, the paper offers a blueprint for building AI agents that are not just powerful but also computationally efficient, addressing the "efficiency" gap between biological and artificial intelligence.
• Generalizability: The framework is domain-agnostic, capable of handling both simple perceptual choices and complex multi-step planning, suggesting a unified mechanism for cognitive control across diverse tasks.

In summary, the paper argues that learning to reason is learning to learn from the output of one's own cognitive operations, and that this process can be effectively modeled and understood through the lens of meta-reinforcement learning in recurrent neural networks.