🤖 machine learning

Data Deletion Can Help in Adaptive RL

This paper demonstrates that randomly deleting a fraction of training data in contextual reinforcement learning improves context estimation and policy robustness by implicitly weighting recent, distribution-aligned samples, a phenomenon theoretically justified by showing that such deletion reduces expected test loss under distribution mismatch when regularization and signal-to-noise ratios fall within specific ranges.

Original authors: Param Budhraja, Aditya Gangrade, Alex Olshevsky, Venkatesh Saligrama

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Param Budhraja, Aditya Gangrade, Alex Olshevsky, Venkatesh Saligrama

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to walk. You train it in a simulation where the floor is sometimes slippery, sometimes sticky, and sometimes has different gravity. The robot learns to adapt to these changes by looking at its recent steps and guessing, "Ah, the floor must be slippery right now," and then adjusting its walk.

This paper introduces a surprising trick to make that robot learn better: Delete some of its training memories.

Here is the breakdown of how this works, using simple analogies:

1. The Problem: The Robot Gets Confused by Old News

In standard training, the robot collects thousands of "memories" (data points) over many rounds.

Early rounds: The robot is clumsy. It makes mistakes and collects data based on bad guesses.
Later rounds: The robot gets smarter. It collects data based on better guesses.

The problem is that when the robot tries to learn how to guess the environment (the "context"), it looks at all its memories equally. It's like trying to learn how to drive a car in the rain by studying a mix of:

Videos of a professional driver in perfect weather (great data).
Videos of a toddler learning to walk in a blizzard (bad data).

The "bad data" from the early, clumsy rounds doesn't match the "real world" the robot will face later. It confuses the learning process.

2. The Solution: The "Fading Memory" Trick

The authors propose a simple, counter-intuitive rule: After every training session, randomly throw away a chunk of the robot's memory bank.

Think of this like a rotating shelf of books:

Every time you add a new book (new data), you randomly pull a few old books off the shelf and throw them in the trash.
Because you do this every round, the oldest books (the clumsy, early data) are the most likely to be thrown away.
The newer books (the smart, recent data) have a better chance of staying.

Why is this magic?

It keeps the "fresh" data: The robot focuses on what it learned recently, which is more relevant to the current situation.
It keeps the "variety": Unlike a system that only keeps the newest data (which might be too narrow), this random deletion keeps a mix of different scenarios, just with less "noise" from the very old, useless attempts.

3. The Results: Small Brains Can Beat Big Brains

The researchers tested this on computer simulations of robots walking and balancing (like a lunar lander or a running ant).

The Surprise: They found that a tiny, simple brain (a small neural network) using this "deletion trick" could actually beat a giant, complex brain (a large neural network) that didn't use the trick.
The Numbers: In some cases, the small model with deletion was 30% better at adapting than the big model without it. Even on average, it improved performance by about 6%.

It's as if a student with a small notebook who only keeps their best, most recent notes outperforms a student with a massive library full of outdated, confusing textbooks.

4. The Theory: Why Does Throwing Away Help?

The authors did some math to explain why this works.

Imagine you are trying to find the center of a target.
If your training data (the arrows you've shot) comes from a slightly different angle than your actual target (a "distribution mismatch"), keeping every single arrow might pull your aim in the wrong direction.
By randomly deleting some arrows, you accidentally remove the ones that are pulling you off-course the most.
The math shows that if the "noise" in your data is high enough (like a lot of wind blowing your arrows), deleting a few random arrows actually helps you hit the bullseye more often.

Summary

The paper argues that in the world of AI that needs to adapt to changing environments, less is more. By randomly deleting old, potentially confusing training data, the AI focuses on what matters most: recent, diverse experiences. This allows even small, simple AI models to become highly adaptable and outperform much larger, more complex models.

1. Problem Statement

The paper addresses the challenge of Adaptive Reinforcement Learning (RL) in real-world deployment. Unlike standard RL where training and testing environments are identical, adaptive RL requires an agent to generalize to unseen but similar environments.

Framework: The authors utilize the Contextual Markov Decision Process (cMDP) framework. A cMDP is a family of MDPs indexed by a low-dimensional context $c$ (e.g., gravity, mass, friction) that affects transition dynamics.
The Challenge: The true context $c$ is known during training but unknown at test time.
Standard Approach: The problem is typically decomposed into two parts:
1. Universal Policy ( $K$ ): Trained to take optimal actions given the state and the true context.
2. Context Estimator ( $\phi$ ): Trained to estimate the context $\hat{c}$ based on observed state-action trajectories.
3. Adaptive Policy: Defined as $\pi = K(\hat{c})$ .
The Issue: In multi-round training, data is collected using progressively better policies. Older trajectories are generated by suboptimal policies and come from a distribution that differs significantly from the deployment scenario (where the estimator is imperfect). This distribution mismatch between training data and deployment reality degrades the performance of the context estimator.

2. Methodology: Random Data Deletion

The authors propose a simple, counterintuitive technique called Random Data Deletion to improve the context estimator's performance.

The Algorithm:
1. Train a universal policy $K$ and an initial context estimator $\phi$ .
2. Collect trajectories using the current adaptive policy $K(\phi)$ .
3. Add these trajectories to a data buffer.
4. Crucial Step: After each training round, randomly delete a fraction $(1 - \alpha)$ of the trajectories currently in the buffer.
5. Retrain the estimator $\phi$ on the remaining data.
Mechanism of Action:
- Since data is collected over multiple rounds, older data corresponds to earlier, less capable policies.
- Random deletion creates an implicit exponential decay on older data. Trajectories from early rounds have a lower probability of survival than those from recent rounds.
- This effectively filters out "stale" data (which causes distribution mismatch) while preserving data diversity (unlike simply keeping only the newest data, which might lack coverage).
Theoretical Justification:
- The authors analyze this via Regularized Empirical Risk Minimization (ERM) with a mismatch between the training distribution ( $P_{train}$ ) and test distribution ( $P_{test}$ ).
- They prove that under mild conditions (specifically for convex loss functions), removing a single uniformly random data point decreases the expected test loss if the distribution mismatch is significant.
- For Ridge Regression, they derive a quantitative condition: Deletion is beneficial when the Signal-to-Noise Ratio (SNR) is sufficiently low and the regularization coefficient is moderate. The SNR threshold directly measures the required magnitude of distribution mismatch for deletion to be helpful.

3. Key Contributions

Novel Technique: Introduction of Random Data Deletion as a method to enhance adaptive RL policies.
Empirical Validation: Demonstration of performance gains across Multi-Layer Perceptrons (MLPs) and Recurrent Neural Networks (RNNs like LSTM/GRU).
Efficiency: Showing that data deletion allows smaller models (e.g., an MLP with 5x fewer parameters) to outperform larger, wider models trained without deletion.
Theoretical Analysis: Providing a theoretical framework explaining why deletion helps, linking it to distribution mismatch and regularization in convex optimization settings.

4. Experimental Results

The authors evaluated the method on Classical Control (Pendulum, LunarLander, Acrobot, MountainCar) and Brax Locomotion (Ant, HalfCheetah) benchmarks, varying contexts like gravity, mass, and friction.

Performance Gains:
- MLPs: Reduced the robustness gap (distance to the optimal "true context" policy) by up to 30%.
- RNNs (LSTM/GRU): Reduced the robustness gap by an average of 6%.
Model Efficiency:
- A Narrow MLP (8,322 parameters) trained with data deletion outperformed a Wide MLP (47,426 parameters) trained without deletion.
- RNNs (LSTM/GRU) generally outperformed the Wide MLP, despite having significantly fewer parameters.
Ablation Studies:
- Compared against Stale Deletion (removing only the oldest data) and Uniform Deletion (randomly sampling for training but keeping all data).
- Random Deletion was superior for RNNs, while Uniform Deletion worked best for Narrow MLPs. Stale deletion performed poorly, confirming that some older data is necessary for diversity.
- Optimal data retention ( $\alpha$ ) was found to be around 0.8 (deleting 20%) or 0.5 (deleting 50%) depending on the architecture and environment.

5. Significance and Conclusion

This work challenges the conventional wisdom that "more data is always better" in RL. It demonstrates that in adaptive settings, older data can be detrimental due to distribution shifts caused by the evolving policy.

Practical Impact: The method offers a computationally cheap, hyperparameter-tunable way to improve RL generalization without changing the underlying architecture or reward function.
Theoretical Insight: It bridges the gap between practical heuristics and theoretical understanding, showing that data deletion acts as a regularizer against distribution mismatch.
Broader Application: The findings suggest that in any learning scenario where the training distribution drifts away from the deployment distribution (common in online learning and sim-to-real transfer), selectively removing stale data can improve generalization.

In summary, the paper establishes that randomly deleting a fraction of training data is a powerful, simple, and theoretically grounded strategy for enhancing the robustness and efficiency of adaptive RL agents in time-varying environments.