🤖 machine learning

Trace-Mediated Peak Bias: Bridging Temporal Credit Assignment and Cognitive Heuristics in Deep Reinforcement Learning

This paper identifies "Trace-Mediated Peak Bias" (TMPB), a systematic failure in deep reinforcement learning where intermediate eligibility traces cause agents to irrationally prioritize high-magnitude reward peaks over cumulative returns due to unnormalized gradient shocks, offering a mechanistic explanation for the human Peak-End Rule and demonstrating that adaptive optimizers are theoretically necessary to mitigate this pathology.

Original authors: Viktor Veselý, Aleksandar Todorov, Erwan Escudie, Matthia Sabatelli

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Viktor Veselý, Aleksandar Todorov, Erwan Escudie, Matthia Sabatelli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to choose between two different paths to get to a treasure. This is the core of Reinforcement Learning: an agent learns by trying things and remembering what worked.

This paper discovers a weird glitch in how these robots learn, which the authors call Trace-Mediated Peak Bias (TMPB). It turns out, this glitch makes robots act surprisingly like humans when they remember their past experiences.

Here is the breakdown of what they found, using simple analogies:

1. The Setup: Two Paths, One Choice

The researchers created a simple game with two paths starting from the same point:

The "Steady" Path: You get a small, consistent reward (like getting $2 every day for 10 days). In total, this path is worth a lot of money.
The "Peak" Path: You get almost nothing, except for one huge, exciting reward in the middle (like getting $10 on day 3) and a decent reward at the very end. In total, this path is worth less money than the Steady path.

The Rational Choice: A perfectly logical calculator would pick the Steady path because the total money is higher.

2. The Glitch: The "Highlight Reel" Effect

When the researchers used a standard learning method (called SGD with "eligibility traces"), the robot made a mistake. It started preferring the Peak path, even though it was worth less money overall.

Why?
The robot's memory system works like a "highlight reel."

Eligibility Traces are like a mental sticky note that says, "Remember what you did a few steps ago, because it might have caused this reward."
When the robot hit that huge $10 "Peak" reward, it created a massive "shock" to its memory. Because the learning system wasn't smart enough to balance this out, that single big moment completely overwrote the memory of all the smaller, steady rewards.

The robot became "irrational." It forgot the total value and only remembered the most intense moment.

3. The Human Connection: The "Peak-End" Rule

The paper points out that this isn't just a robot bug; it's a human bug too.

In psychology, there is a famous idea called the Peak-End Rule. It says that when humans remember an experience (like a vacation or a movie), we don't remember the total amount of fun we had. Instead, we judge it based on the most intense moment (the Peak) and the very last moment (the End).
The paper shows that the robot's glitch is a mathematical version of this human bias. The robot, just like a human, got "hijacked" by the intensity of the peak moment and ignored the boring, steady reality.

4. The Fix: The "Smart" Teacher

The researchers found a way to stop the robot from making this mistake. They switched the learning method from a "fixed-step" teacher to an adaptive optimizer (like RMSprop).

The Old Way (Fixed): If you get a huge reward, the teacher yells "Change everything!" and the robot panics, overwriting its whole memory.
The New Way (Adaptive): This teacher is smarter. It sees the huge reward and says, "Okay, that was a big shock, but let's not freak out. Let's adjust our memory carefully so one big moment doesn't erase the whole story."

When they used this "smart teacher," the robot stopped being irrational. It correctly chose the Steady path every time, realizing that the total value mattered more than the single highlight.

The Big Takeaway

The paper argues that "irrational" human behaviors (like judging a whole vacation by its best day) might not be a flaw in our brains, but a natural result of how our brains process rewards. If our brains use a simple "shock-based" learning system without a "smart filter" to balance those shocks, we will naturally develop these biases.

For Artificial Intelligence, the lesson is clear: If you want your AI to be truly rational and not fall for "highlight reel" traps, you must use the smart, adaptive learning tools that normalize these big shocks. Otherwise, your AI will inherit human-like irrationalities naturally.

Technical Summary: Trace-Mediated Peak Bias in Deep Reinforcement Learning

Problem Statement
Temporal credit assignment—the process of determining which past states are responsible for current rewards—is a foundational challenge in both biological and artificial intelligence. While eligibility traces (e.g., in TD( $\lambda$ ) algorithms) effectively bridge temporal gaps in tabular settings, their interaction with non-linear function approximation in deep reinforcement learning (RL) remains poorly understood. The paper identifies a systematic failure mode in deep RL termed Trace-Mediated Peak Bias (TMPB). In this phenomenon, agents equipped with eligibility traces and non-linear neural networks irrationally prefer trajectories containing high-magnitude reward "peaks" over alternatives with higher cumulative returns. This behavior mirrors the human Peak-End Rule, a cognitive heuristic where retrospective evaluations are dominated by the most intense moments of an experience rather than its integrated utility. The core problem is that standard optimization methods fail to normalize the "gradient shocks" caused by distal, high-saliency rewards, leading to global value overestimation and sub-optimal policy selection.

Methodology
To characterize this interaction, the authors designed a stylized episodic Markov Decision Process (MDP) called the Two-Door Environment. The environment features two distinct paths of length $T=10$ originating from a common state $S_0$ :

$\tau_{steady}$ : Provides a constant reward of $r_t = 2$ at every step.
$\tau_{peak}$ : Provides a sparse reward profile with a high-magnitude "peak" ( $r_3 = 10$ ), a concluding reward ( $r_{10} = 6$ ), and zero elsewhere.

Given a discount factor $\gamma = 0.95$ , the rational optimal values are $V^*(S_0|\tau_{steady}) \approx 16.05$ and $V^*(S_0|\tau_{peak}) \approx 12.81$ . The authors evaluated value estimates using a neural network with one hidden layer of 16 ReLU units. Training was conducted over 1,000 episodes using a uniform behavior policy and Stochastic Gradient Descent (SGD) with a fixed learning rate. The loss function utilized the forward-view equivalent of the gradient-based eligibility trace:
$\mathcal{L}(\theta) = \sum_{t=0}^{T-1} \lambda^t (V_\theta(s_t) - [r_t + \gamma V_\theta(s_{t+1})])^2$
The study compared the performance of fixed-step-size SGD against adaptive optimizers (specifically RMSprop) across varying trace depths ( $\lambda$ ).

Key Results
The experiments revealed a systematic overestimation of the value of the "Peak" trajectory ( $V(S_0|\tau_{peak})$ ) when using SGD with intermediate trace depths ( $0.15 < \lambda < 0.50$ ). In this "Irrationality Zone," the value estimation for the inferior path erroneously surpassed that of the superior "Steady" path, causing the agent to prefer the sub-optimal trajectory.

Conversely, when the same setup was trained using the adaptive optimizer RMSprop, this bias was mitigated, and the agent correctly identified the "Steady" path as optimal. The results demonstrate that the bias is not inherent to the trace mechanism itself but is a specific artifact of how fixed-step-size optimizers handle the variance introduced by high-magnitude rewards in a distributed neural system.

Mechanistic Account
The paper provides a mechanistic explanation for TMPB:

Gradient Shocks: High-magnitude rewards generate large Temporal Difference (TD) errors ( $\delta_t$ ). In the presence of eligibility traces, these errors are propagated back through time, creating "gradient shocks" ( $\Delta\theta_t = \alpha \delta_t z_t$ ).
Lack of Normalization: In fixed-step-size SGD, the parameter shift is directly proportional to the magnitude of the reward. Consequently, a single peak update can disproportionately reconfigure shared weights, effectively "hijacking" the neural manifold and exceeding the integrated sum of steady, lower-magnitude updates.
Adaptive Mitigation: Adaptive optimizers like RMSprop scale updates by the inverse of the second moment of gradients. This normalization dampens the impact of high-variance outliers (the peak rewards), preventing the gradient shocks from distorting the global value estimation.

Significance and Contributions
The paper makes three primary contributions:

Identification of TMPB: It formally identifies and defines Trace-Mediated Peak Bias as a systematic failure mode in deep RL where agents exhibit "irrational" preferences for high-saliency, low-cumulative-reward trajectories.
Computational Parallel to Cognitive Bias: It demonstrates that TMPB serves as a direct computational parallel to the psychological Peak-End Rule. This suggests that certain human-like "irrationalities" may be emergent properties of the mathematical constraints of credit assignment in distributed systems, rather than purely biological heuristics.
Theoretical Necessity of Adaptive Optimization: The authors argue that adaptive optimization is a theoretical necessity for rational value estimation in non-linear systems. Without variance-based normalization, systems utilizing distributed representations and trace-like credit assignment are structurally prone to saliency distortions.

Conclusion
The work reframes the relationship between cognitive psychology and reinforcement learning. While RL historically drew inspiration from psychology, this paper posits that the mathematical pathologies of modern deep RL now offer a rigorous language for explaining human "irrationalities." The findings suggest that human saliency distortions may reflect a biological reliance on simple gradient-based updates that lack the sophisticated normalization found in adaptive artificial systems. For AI development, this highlights that "human-like" neural architectures naturally inherit human-like vulnerabilities, making the choice of optimizer a fundamental theoretical concern rather than a mere hyperparameter tuning task.