🤖 machine learning

Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learning

This paper investigates how to optimize replay retention in continual model-based reinforcement learning under dynamics shifts by characterizing the trade-off between change magnitude and transition age, demonstrating that an estimator can effectively guide whether to keep or discard old data based on whether dynamics changes are permanent or recurring.

Original authors: Everest Yang, Skye Thompson, George D. Konidaris

Published 2026-09-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Everest Yang, Skye Thompson, George D. Konidaris

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots that learn to move often rely on a mental model of how their bodies work. They try an action, watch what happens, and use that experience to predict the future, gradually refining their movements until they can walk, run, or carry objects with ease. This process, known as reinforcement learning, is powerful because it allows machines to adapt to new tasks without being explicitly programmed for every possible situation. However, a robot's world is rarely static. A leg might break, a payload might change, or the ground might become slippery. When these physical changes occur, the robot's old memories of how it used to move can become misleading. If the robot keeps training on these outdated experiences, it learns the wrong lessons and struggles to adapt. If it throws away all its old data to start fresh, it loses valuable information that might still be useful, or that might become necessary again if the robot returns to its original state. The central challenge is knowing exactly when to hold onto the past and when to let it go.

Researchers at Brown University investigated this precise dilemma by studying how robots should manage their memory banks when their physical dynamics shift. They focused on a specific type of learning where the robot maintains a history of its interactions, called a replay buffer, to train its internal model. The team wanted to determine if a robot should always keep its entire history of experiences, or if it should strictly limit its training to only the most recent data. To find the answer, they simulated a robot named Walker, a two-legged machine, and introduced different types of physical changes. In some scenarios, the robot suffered a permanent injury, such as a motor losing half its strength. In others, the robot experienced recurring changes, where the damage would appear and then disappear in a cycle, mimicking a situation where a robot picks up a heavy load and then sets it down. They also tested a control group where the robot's body remained exactly the same throughout the training.

The results revealed a clear pattern that depends entirely on the nature of the change. When the robot faced a permanent shift, such as a broken motor, the strategy of keeping only the newest data worked best. By discarding the old, mismatched memories, the robot was able to learn the new reality much faster and achieve higher performance scores. In these cases, the older data was simply noise that slowed down the learning process. However, the situation flipped completely when the changes were recurring. When the robot's dynamics returned to their original state, the strategy of keeping only recent data became a liability. The robot had thrown away the very memories it needed to remember how to move correctly, causing its performance to drop significantly. In these recurring scenarios, keeping the full history of experiences allowed the robot to recover quickly whenever the old conditions returned.

The researchers found that the decision to forget or remember hinges on two specific factors. The first is the size of the change. Small shifts in how the robot moves do not justify throwing away old data, because the old experiences are still largely relevant. Large, permanent changes, however, make the old data so inaccurate that it is better to start with a clean slate. The second factor is the predictability of the change. If the robot can expect the old conditions to return, holding onto the past is essential. The team developed a method to estimate these factors using only the data the robot generates while moving, without needing to know the internal physics of the simulation. By analyzing how the robot's motors responded to commands, they could detect when a change had occurred and estimate how severe it was. This allowed them to make an informed choice about whether to keep the old history or switch to a fresh, short-term memory.

In their experiments, the researchers tested these ideas across different robot shapes and learning algorithms to ensure the findings were robust. They confirmed that the benefits of discarding old data after a permanent injury were consistent, while the costs of doing so when conditions were cyclical were equally consistent. They also compared their approach to other methods, such as keeping a random sample of old data or using complex weighting systems, and found that the simple rule of "keep recent data for permanent changes, keep history for recurring ones" was often the most effective. The study suggests that for robots operating in the real world, where damage is permanent but environmental conditions might fluctuate, the ability to judge the type of change is just as important as the ability to learn. By using the robot's own movement data to decide what to remember, machines can become more resilient, adapting quickly to injury without forgetting how to function when the injury is temporary.

Technical Summary: Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learning

Problem Statement
In model-based reinforcement learning (RL), agents learn world models from replayed experience to adapt to continuous control tasks. However, when a robot's dynamics shift (e.g., due to actuator damage, payload changes, or environmental perturbations), the utility of previously collected transitions becomes ambiguous. Training on outdated ("stale") transitions can slow adaptation, yet discarding them risks losing valuable data if the original dynamics return (recurring changes). The core problem is determining when a replay strategy should prioritize recent data (recency) versus retaining the full history (passive), particularly when ground-truth labels for data staleness are unavailable in real-world deployments.

Methodology
The authors investigate this trade-off using two locomotion morphologies (Walker and Cheetah) and two model-based RL algorithms (DreamerV3 and TD-MPC2). They evaluate performance under three dynamic regimes:

Permanent Changes: A lasting shift in dynamics (e.g., reduced actuator gain or damage).
Recurring Changes: Dynamics that shift and then return to their original state.
No-Change Control: A baseline with no dynamics shift.

The study compares a Passive strategy (sampling from a full buffer of 1 million transitions) against a Recency strategy (restricting replay to the newest 10,000 or 20,000 transitions). To characterize the trade-off, the authors define two key metrics:

Change Magnitude ( $M$ ): The size of the shift in underlying dynamics relative to older transitions.
Age–Staleness AUC: The area under the receiver operating characteristic curve measuring how well transition age separates stale data from fresh data. An AUC of 1.0 implies perfect separation (all stale data is older than all fresh data), while 0.5 implies random separation.

To address the lack of ground-truth labels in real-world scenarios, the authors propose a Retrospective Estimator. This system uses interaction data (specifically actuator responses derived from joint velocities and actions) to:

Detect regime changes using Pruned Exact Linear Time (PELT) change-point detection.
Estimate the change magnitude by comparing the median actuator response of stale versus fresh segments.
Estimate the age–staleness AUC based on the detected segments.

These estimates feed into a selector (Algorithm 1) that decides whether to switch to a recency-based window or retain the full history.

Key Contributions

Characterization of Replay Retention: The paper establishes that the optimal replay strategy depends on both the magnitude of the dynamics change and the temporal structure of the change (specifically, the relationship between transition age and staleness).
Context-Dependent Forgetting: The authors demonstrate that the same dynamics change can favor forgetting (recency) if the change is permanent, but favor retention (passive) if the dynamics recur. This indicates that change magnitude alone is insufficient for selecting a replay strategy; the recurrence pattern is critical.
Retrospective Estimation: The paper presents a method to detect dynamics changes and estimate the necessary quantities (magnitude and age-staleness AUC) using only actuator response data, without access to simulator parameters or ground-truth dynamics labels.

Results

Permanent vs. Recurring Dynamics: Under permanent changes (e.g., gain reduction to $g=0.5$ or actuator damage), the Recency strategy significantly outperformed Passive (e.g., +146 return improvement for Walker). Conversely, under recurring dynamics, Recency severely underperformed Passive (e.g., -191 return), as discarding data that later became relevant was costly.
Magnitude Thresholds: The benefit of forgetting is contingent on change magnitude. Small changes (e.g., $g=0.7$ ) did not justify discarding older data, while larger changes (e.g., $g=0.5$ ) did. The transition point where Recency becomes beneficial was identified between $g=0.6$ and $g=0.5$ .
Window Size and Buffer Capacity: Increasing the recency window size reduced the performance loss under recurring dynamics, confirming that retaining more historical data is beneficial when dynamics are likely to return. A reservoir buffer (uniform sampling of history) did not replicate the gains of recency after permanent changes, indicating that the benefit comes from the specific exclusion of stale data, not just reduced capacity.
Generalization: The trends held across different morphologies (Cheetah) and algorithms (TD-MPC2). In the Real-World RL (RWRL) benchmark, Recency performed poorly under cyclic damping (recurring) but showed mixed results under drifting friction.
Estimator Performance: The retrospective actuator-response estimator successfully detected permanent changes within three episodes. In a retrospective evaluation across 24 runs, the selector chose the better-performing strategy in 19 cases. However, the estimator struggled to accurately recover AUC for recurring endpoints, and in some cases, the decision relied on the magnitude estimate falling below the threshold rather than accurate AUC estimation.

Significance and Claims
The paper claims that replay retention is not a static optimization but a dynamic decision dependent on how the environment evolves. The authors emphasize that change magnitude alone is not enough to select an effective replay strategy; one must also predict whether older data will become useful again (recurrence).

The significance of the work lies in providing a framework for adaptive replay selection that can operate without privileged information. By showing that actuator response alone can estimate the necessary quantities to guide replay decisions, the authors suggest that these signals could enable real-world robots to manage their experience buffers autonomously. The paper remains modest, noting that the current estimator is retrospective and that an online version capable of making decisions based on partial trajectories remains a direction for future work. The findings highlight that aggressive age-based forgetting is beneficial primarily after large, permanent shifts, whereas retaining a broader history is crucial when dynamics are non-stationary and recurrent.

Technical Summary: Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learning

More like this