🧬 biology

Reward function compression facilitates goal-dependent reinforcement learning

This paper proposes and experimentally validates that humans enhance reinforcement learning efficiency by initially relying on working memory to compress complex, abstract goals into simplified, stable reward functions that transfer to long-term memory, thereby freeing cognitive resources for automatic evaluation.

Original authors: Gaia Molinaro, Anne G. E. Collins

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gaia Molinaro, Anne G. E. Collins

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Idea: Learning with a "Mental Backpack"

Imagine your brain has a backpack called Working Memory. This backpack is great for holding things you need right now, like a phone number you just heard or a rule you just read. But it has a strict limit: it can only hold about 3 to 4 items at once. If you try to stuff too much in, things fall out, or you get so overwhelmed you can't think clearly.

The paper argues that when humans try to learn new things based on abstract goals (like "win this game" or "collect the purple card"), we initially have to carry the rules for those goals inside this small backpack. This is cognitively expensive. It's like trying to solve a math problem while simultaneously juggling three heavy bowling balls. You can do it, but you'll be slower and make more mistakes.

The authors propose a clever solution: Reward Function Compression.

Think of this as folding a map.

The Uncompressed Map: Imagine a map of a city where every single street, building, and tree is drawn in high detail. It's accurate, but it's huge and hard to carry in your backpack.
The Compressed Map: After you've visited the city a few times, you realize you don't need every detail. You can fold the map down to just the main highways and the location of the park. It's tiny, fits easily in your pocket, and you can pull it out instantly without thinking.

The paper suggests that our brains do this with goals. At first, we remember every specific detail of what a "win" looks like. But with practice, we compress that information into a simple, automatic rule (e.g., "All purple things are good"). Once this rule is "compressed" and moved to Long-Term Memory (a giant warehouse outside the backpack), we don't have to carry it in our backpack anymore. This frees up mental space to actually learn the task faster.

The Experiments: How They Tested This

The researchers ran six experiments using a computer game to prove this idea. Here is how they did it:

1. The Game Setup

Participants had to learn which key to press for which picture.

The Easy Way (Points): They got standard feedback like "+1" or "+0". This is like getting a score.
The Hard Way (Goals): Instead of numbers, they saw abstract fractal images labeled "Goal" (Good) or "Nongoal" (Bad). They had to figure out which picture was the "Goal" and press the right key to get it.

2. Experiment 1: The "Too Many Goals" Test

The Setup: In some rounds, there was only one "Goal" image to look for. In other rounds, there were four different "Goal" images that changed.
The Result: When there was only one goal, people learned quickly. When there were four different goals, their learning slowed down significantly.
The Takeaway: This proves that our "mental backpack" gets overloaded. Trying to keep track of four different rules at once uses up all the space needed for learning.

3. Experiment 2: The "Compressible vs. Uncompressible" Test

This was the most important experiment. The researchers kept the number of goals the same (so the backpack load was the same), but they changed the structure of the goals.

Compressible Goals: Imagine the "Goal" images were all Red Squares, and the "Nongoal" images were all Blue Triangles. Even though there were many images, the rule was simple: "If it's Red, it's a Goal." This is easy to compress.
Uncompressible Goals: Imagine the "Goal" images were a weird mix: a Red Square, a Blue Circle, a Green Triangle, etc. There was no simple rule. You had to memorize every single specific image.
The Result: People learned much faster in the Compressible condition. Even though they had to remember the same number of images, they could "fold the map" into a simple rule ("Red = Good"). In the Uncompressible condition, they couldn't fold the map, so they had to carry the whole heavy thing in their backpack, slowing them down.

4. The Speed Connection

The researchers also measured how fast people pressed a button to "collect" the good outcome.

Finding: The faster someone was at recognizing the "Goal" image, the better they were at learning the game.
Why? This suggests that when the brain processes the reward automatically (because the rule is compressed), it frees up energy for the actual learning. If you are still struggling to figure out "Is this a goal?", you have less brainpower left to learn the rules of the game.

What This Means (According to the Paper)

The paper concludes that human learning isn't just about memorizing facts. It's about efficiency.

Flexibility has a cost: We can set any goal we want (even abstract ones), but it takes a lot of mental energy at first.
Compression is the key: To become efficient, our brains must turn complex, high-dimensional goals into simple, low-dimensional rules.
Automaticity: Once that rule is compressed and stored in long-term memory, we stop "thinking" about the goal and start "feeling" it automatically. This frees up our working memory to learn new things faster.

In short: We learn best when we can turn a complex list of "what to do" into a simple, automatic habit. If the rules are too messy to simplify, our brains get tired, and learning suffers.

Technical Summary: Reward Function Compression Facilitates Goal-Dependent Reinforcement Learning

Problem Statement

Reinforcement learning (RL) in humans extends beyond primary and secondary rewards to include flexible, goal-dependent outcomes where novel, abstract stimuli are imbued with value based on current objectives. While this flexibility allows humans to pursue diverse goals, it is cognitively costly and often reduces learning efficiency. The precise cognitive mechanisms by which humans align their value systems to current goals, and why this alignment is difficult to maintain, remain uncharacterized. Specifically, it is unclear how the brain manages the trade-off between the flexibility of assigning value to novel outcomes and the efficiency of learning from them. Previous work suggests that goal-dependent learning relies on working memory (WM), but the specific process by which this reliance transitions to more automatic, efficient processing is unknown.

Methodology

The authors conducted six experiments (four primary and two supplementary) using a modified RL task where participants learned stimulus-response mappings via trial and error.

Experimental Design

Task Structure: Participants viewed six stimuli and had to learn the correct key press for each. Feedback was deterministic.
Conditions:
- Points Blocks (Baseline): Feedback consisted of standard numeric points ("+1" for correct, "+0" for incorrect).
- Goals Blocks: Feedback consisted of abstract fractal images labeled "Goal" or "Nongoal." Participants had to learn which actions led to the "Goal" image.
Reward Collection: In all blocks, participants were required to press a separate key to "collect" desirable outcomes (points or goal images) and ignore undesirable ones. Reaction times (RTs) for this collection phase were recorded as a behavioral proxy for reward processing efficiency.

Specific Manipulations

Experiment 1 (WM Load): Varied the number of unique Goal/Nongoal image pairs per block (1, 2, 3, or 4 pairs). The total number of trials remained constant, but the size of the goal space increased, increasing the WM load required to maintain the goal rule.
Experiment 2 (Compressibility): Kept the number of unique goal images constant (8 images total) but manipulated the rule complexity.
- Compressible: A single feature (e.g., color) distinguished all Goal images from Nongoal images.
- Incompressible: All three features (shape, color, fill) were required to distinguish Goal from Nongoal (a conjunctive rule).
Supplementary Experiments (3–6): These re-analyzed previous data or introduced new controls to rule out alternative mechanisms, such as simple repetition effects, top-down value attribution without compression, or interference from inconsistent labeling.

Computational Modeling

The authors fitted a family of computational models based on a standard RL architecture (delta rule for value updates, softmax policy for action selection).

Base Model: Included parameters for learning rate ( $\alpha$ ), negative outcome multiplier ( $\zeta$ ), noise ( $\epsilon$ ), forgetting ( $\phi$ ), and stickiness ( $\eta$ ).
Advanced Models: Tested whether specific parameters (learning rate, noise, stickiness) varied by condition.
RT-Conditioned Models: Crucially, the authors tested models where the learning rate was directly scaled by the median log reward collection RT for that specific condition. This tested the hypothesis that faster reward processing (indicating compression/automation) frees cognitive resources to improve learning rates.

Key Results

Behavioral Findings

Goal-Dependent Learning Cost: Learning performance was significantly lower in Goals blocks compared to Points blocks, even in the simplest single-goal condition, confirming the cognitive cost of abstract goal attribution.
WM Load Effect (Exp 1): Learning accuracy decreased parametrically as the number of unique Goal/Nongoal pairs increased (from 1 to 4). This suggests that maintaining a larger goal space in WM impairs the learning process.
Compression Effect (Exp 2): When the goal space was held constant but the rule complexity varied, learning accuracy was significantly higher in Compressible blocks than in Incompressible blocks. This indicates that performance gains are driven by the ability to form a simplified rule, not merely by repetition.
Reward Collection RTs: Faster reward collection RTs strongly correlated with better learning performance across conditions. This correlation was specific to reward collection RTs and not choice RTs, suggesting that the efficiency of evaluating the outcome (reward processing) is the bottleneck, not the decision-making process itself.
Supplementary Findings:
- Repetition alone (Exp 5) did not improve learning if the goal space exceeded WM capacity.
- Inconsistent labeling across trials (Exp 6) specifically impaired learning, supporting the need for a stable rule to be compressed.

Computational Modeling Findings

Value Updating Bottleneck: The best-fitting models indicated that the primary impact of goal-dependent learning was on the value updating stage (learning rate), not on choice selection noise.
RT-Conditioned Learning Rate: A model where the learning rate was a function of reward collection RTs provided a better and more parsimonious fit to the data than models with free parameters for each condition or models where RTs affected choice noise.
Interpretation: This supports the hypothesis that efficient reward processing (via compression) frees up executive resources, allowing for more effective updating of stimulus-action associations.

Key Contributions

Reward Function Compression: The paper proposes and empirically validates a mechanism where complex, high-dimensional goal information is compressed into a simplified, stable rule (a "compressed reward function") that can be transferred from working memory to long-term memory.
Mechanism for Efficiency: It demonstrates that this compression process is the key factor that alleviates the cognitive load of goal-dependent learning, thereby boosting learning efficiency.
Behavioral Marker: It identifies reward collection reaction time as a specific behavioral signature of reward function compression and processing efficiency.
Computational Formalization: It provides a computational model linking reward processing efficiency (RT) directly to learning rates, offering a formal account of how executive functions interact with RL.

Significance

The authors claim that these findings illuminate the cognitive mechanisms underlying intrinsic motivation and goal-directed learning. By showing that the brain manages the trade-off between flexibility and efficiency through reward function compression, the study offers a potential framework for understanding why some goals are harder to achieve than others. The authors suggest these insights could inform behavioral interventions and educational strategies to support human goal achievement, particularly by helping individuals structure goals in ways that facilitate compression and automation. The paper remains agnostic regarding the specific neural implementation of this compression but posits that it relies on the interplay between working memory and long-term memory systems.