💻 computer science

Continual Domain Randomization

This paper proposes Continual Domain Randomization (CDR), a novel approach that combines domain randomization with continual learning to sequentially train reinforcement learning policies on subsets of simulation parameters, thereby overcoming the sub-optimality of simultaneous randomization and achieving robust sim-to-real transfer in robotic tasks.

Original authors: Josip Josifovski, Sayantan Auddy, Mohammadhossein Malmir, Justus Piater, Alois Knoll, Nicolás Navarro-Guerrero

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Josip Josifovski, Sayantan Auddy, Mohammadhossein Malmir, Justus Piater, Alois Knoll, Nicolás Navarro-Guerrero

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to pick up a cup. The smartest way to do this is usually to let the robot practice millions of times in a computer simulation first, so it doesn't break the real robot or hurt anyone. But here's the problem: a computer simulation is never perfectly like the real world. The real robot might be slightly heavier, the motors might be a bit slower, or the sensors might be a little "noisy." This difference is called the "reality gap." If you train a robot in a perfect simulation, it often fails miserably when you put it in the real world because it learned to rely on the simulation's perfection.

To fix this, scientists use a trick called Domain Randomization. Instead of trying to make the simulation perfect, they intentionally mess it up. They make the robot heavier in one run, lighter in the next, add fake sensor noise, or delay the controls. The idea is that if the robot learns to handle all these messed-up versions, it will be robust enough to handle the real world, which is just another version of "messy."

The Problem with the Old Way
The traditional method is like trying to learn to swim by jumping into a stormy ocean with strong currents, heavy waves, and cold water all at once on your very first day. It's too hard! The robot gets overwhelmed, confused, and learns a bad strategy (or fails to learn anything at all) because the task is too difficult.

The New Solution: Continual Domain Randomization (CDR)
The authors of this paper propose a smarter way to learn, which they call Continual Domain Randomization (CDR). Think of it like learning to drive a car:

Start Easy: First, you learn to drive in an empty, perfect parking lot with no wind, no other cars, and perfect tires. You get the basics down.
Add One Challenge at a Time: Once you are good at that, you don't suddenly throw you into a hurricane. Instead, you add just one new challenge. Maybe you drive in the rain. Once you master the rain, you add wind. Once you master wind, you add heavy traffic.
Remember Everything: The tricky part is that when you learn to drive in the rain, you shouldn't forget how to drive on dry pavement. This is where the "Continual Learning" part comes in. The robot uses a special memory technique (called Elastic Weight Consolidation) that acts like a "safety net." It allows the robot to learn new skills (like handling rain) without "forgetting" the old skills (driving on dry pavement).

How They Tested It
The researchers tested this idea on two tasks:

Reaching: A robot arm trying to touch a specific spot.
Grasping: A more complex task where a robot arm has to find a box, align its gripper perfectly, and pick it up.

They compared their "step-by-step" method (CDR) against two other approaches:

The "Perfect" Simulator: Training only in a clean, perfect world (which fails in reality).
The "Chaos" Simulator: Throwing all the problems (rain, wind, traffic) at the robot at the same time.
The "Forgetful" Method: Adding problems one by one but not using the memory safety net (so the robot forgets the previous steps).

The Results
The results were impressive:

The robot trained with CDR learned effectively in the simulation.
When they moved the robot to the real world, it performed better or just as well as the robot trained in "Chaos" mode.
Crucially, CDR was more stable. It didn't matter if they added the challenges in a different order (rain first vs. wind first); the robot still learned well. The "Forgetful" method, however, struggled if the order of challenges changed.

In a Nutshell
This paper shows that instead of drowning a robot in a sea of random variables all at once, it's better to teach it step-by-step, adding one difficulty at a time while ensuring it remembers how to handle the previous ones. This creates a robot that is ready for the messy, unpredictable real world without needing to be tested on the real world during training.

1. Problem Statement

Deep Reinforcement Learning (DRL) in robotics often relies on simulated environments to overcome sample inefficiency and safety concerns. However, the "reality gap"—the discrepancy between simulation and the real world—often leads to poor performance when transferring policies (sim2real) without further tuning.

Limitations of Current Approaches:
- Perfect Simulation: Impossible to achieve for complex systems due to hardware wear and environmental changes.
- Domain Adaptation (Finetuning): Requires real-world data, which is time-consuming and potentially unsafe.
- Standard Domain Randomization (DR): Randomizes all simulation parameters simultaneously to cover the reality gap. However, randomizing too many parameters at once increases task complexity and agent uncertainty, often leading to sub-optimal policies or failure to converge.
- Sequential Randomization without CL: Training sequentially on subsets of parameters without retaining knowledge of previous steps leads to catastrophic forgetting, where the agent overfits to the most recent randomization and loses robustness to earlier ones.

2. Methodology: Continual Domain Randomization (CDR)

The authors propose Continual Domain Randomization (CDR), a framework that combines Domain Randomization with Continual Learning (CL) to enable sequential training on subsets of randomization parameters while retaining knowledge from previous steps.

Core Workflow

Initialization: The agent is first trained in an idealized simulation (no randomization) to learn the base task.
Sequential Training: The agent is subsequently trained on a sequence of simulations, where each step enables a specific subset of randomization parameters (e.g., latency, torque, noise).
Continual Learning: To prevent catastrophic forgetting, the system employs regularization-based CL algorithms to "remember" the effects of previous randomizations.

Algorithmic Implementation

The authors implement two variants using Proximal Policy Optimization (PPO) combined with Elastic Weight Consolidation (EWC):

CDR-λ (Offline):
- Uses EWC to regularize the policy network.
- Maintains a separate snapshot of the network parameters ( $\theta^*$ ) and the empirical Fisher Information Matrix (FIM) for every past task.
- Loss Function: $L(\theta) = L_C(\theta) + \sum \frac{\lambda_i}{2} ||\theta - \theta^*_i||^2_{F_i}$
- Trade-off: High memory usage (linear scaling with the number of tasks) but strong retention of past knowledge.
CDR-Oλ (Online):
- Uses an Online-EWC variant to reduce memory overhead.
- Maintains only one snapshot of the network and a single, cumulative Fisher Information Matrix.
- The Fisher matrix is updated recursively: $F^*_i = \gamma F^*_{i-1} + F_i$ .
- Trade-off: Constant memory usage, but potentially weaker retention of very old tasks compared to the offline version.
Normalization: A key technical detail is the normalization of the Fisher Information Matrix elements to the range $[0.0, 1.0]$ after each task. This decouples the regularization constant $\lambda$ from the arbitrary scaling of different tasks, allowing a single $\lambda$ to be used across the entire sequence.

3. Experimental Setup

The authors evaluated CDR on two robotic manipulation tasks using a KUKA LBR iiwa 14 arm:

Reaching Task: A 2-joint arm reaching a target.
Grasping Task: A 7-joint redundant arm with a Robotiq 3-finger gripper grasping a cube.

Randomization Parameters:
Three parameters were randomized sequentially in different orders (e.g., Torque $\to$ Latency $\to$ Noise):

Latency (L): Random delay in state observation.
Torque (T): Randomized joint stiffness and damping.
Noise (N): Sensor noise.

Baselines:

Ideal: Trained in non-randomized simulation.
Randomized: Trained with all parameters randomized simultaneously.
Finetuning: Sequential training without CL (no regularization).

4. Key Results

Reaching Task

Sim2real Performance: The Ideal model performed best in simulation but failed in the real world (high jitter, low reward) due to overfitting. The Randomized baseline performed well in reality but had lower simulation rewards.
CDR Performance: Both CDR variants achieved sim2real transfer comparable to or better than the fully Randomized baseline.
- CDR-Oλ achieved the highest real-world rewards and stability.
- Finetuning performed well in simulation but showed poor transfer to the real world compared to CDR, confirming that forgetting previous randomizations is detrimental.
Order Sensitivity: Sequential baselines (Finetuning) were highly sensitive to the order of randomization. CDR was significantly more robust to the order of parameters.

Grasping Task

Complexity: The grasping task was harder, requiring higher precision.
Stability: CDR strategies showed lower variance and more stable training progress compared to Finetuning.
Real-World Success:
- CDR-Oλ achieved a 78% grasp success rate in simulation and 60% in reality (with NLT ordering), outperforming the Randomized baseline (40% real success).
- Finetuning showed high variance; depending on the order, it could be the best or worst performer, highlighting its instability.
- CDR models provided consistent performance regardless of the randomization order.

Regularization Strength ( $\lambda$ )

Low $\lambda$ : Models behave like Finetuning (forgetting previous tasks), leading to poor real-world transfer.
High $\lambda$ : Models become too rigid, unable to adapt to new randomizations, resulting in poor performance in both simulation and reality.
Optimal Range: A middle range of $\lambda$ provided the best balance, enabling adaptation while retaining robustness.

5. Key Contributions

Novel Framework: Introduction of Continual Domain Randomization (CDR), which treats domain randomization as a continual learning problem to sequentially introduce complexity without catastrophic forgetting.
Memory Efficiency: Development of an Online CDR (CDR-Oλ) variant that avoids the linear memory scaling of traditional EWC by maintaining a single Fisher matrix, making it scalable for long sequences of randomizations.
Robustness to Ordering: Demonstration that CDR is less sensitive to the order in which randomization parameters are introduced compared to standard sequential finetuning.
Zero-Shot Transfer: Successful zero-shot sim2real transfer on real robots without requiring any real-world training data or domain adaptation.

6. Significance and Conclusion

The paper addresses a critical bottleneck in robotic RL: the trade-off between the difficulty of training with high randomization and the risk of overfitting to a specific simulation. By integrating continual learning, CDR allows agents to "learn how to learn" across varying simulation conditions, building a policy that is robust to a wide range of dynamics.

Practical Impact: CDR offers a flexible, general framework for sim2real transfer that does not require defining all randomization parameters upfront or having access to real-world data for fine-tuning.
Future Work: The authors suggest combining CDR with automated or active domain randomization to dynamically determine optimal parameter ranges and extending the approach to vision-based randomization.

In summary, CDR provides a superior alternative to both static randomization and simple sequential finetuning, achieving state-of-the-art sim2real performance in complex robotic manipulation tasks.