Technical Summary: What Matters for Simulation-to-Online Reinforcement Learning on Real Robots

Problem Statement

While Reinforcement Learning (RL) has demonstrated success in robotics, most current systems rely on offline learning within simulators or fixed datasets, avoiding the canonical online learning setting. This approach limits performance to the quality of the priors and fails to adapt to the "open world." Although simulators are imperfect, obtaining high-quality real-world pre-training data is prohibitively expensive compared to domains like language modeling. Consequently, future autonomous systems must learn online through embodied interaction.

However, naively fine-tuning a simulation-trained policy on real hardware often leads to catastrophic forgetting and instability. This paper investigates the specific design choices required to make standard off-policy RL reliable in a simulation-to-online setting, rather than proposing a new algorithm. The core challenge is the "downward spiral": when a policy trained in simulation ($p_0$) is deployed on real dynamics ($p$), the resulting distribution shift causes the critic to become inaccurate. This inaccuracy reinforces poor actions, leading to further distribution shifts and performance collapse.

Methodology

The authors conduct a large-scale empirical study involving 100+ real-world training runs across three distinct robotic platforms:

1. Franka Emika Panda: A vision-based manipulation task (grasping and lifting a cube).
2. Unitree Go1: A quadruped locomotion task following joystick commands.
3. Race Car: A high-speed navigation task involving parking, characterized by fast control loops (60 Hz) and complex tire friction dynamics.

The study utilizes Soft Actor-Critic (SAC) as the baseline off-policy algorithm. Instead of introducing novel algorithmic components, the authors systematically ablate standard design choices to identify a robust "recipe" for stable transfer.

The Proposed Recipe

The paper identifies four critical components that, when combined, enable stable online fine-tuning:

1. Retained Replay (Data Anchoring):

   • Mechanism: Instead of discarding simulation data ($\mathcal{D}_0$) or relying solely on online data, the authors maintain a mixture of simulation and real-world data in the replay buffer.
   • Strategy: They use an annealing schedule where the sampling probability of online data ($\alpha$) starts at 0.5 and gradually increases to 1.0.
   • Rationale: Retaining $\mathcal{D}_0$ anchors the replay distribution, preventing the critic equilibrium from shifting too abruptly due to the sim-to-real gap, thereby stabilizing early adaptation.

2. Warm-Started Online Replay:

   • Mechanism: Before any policy updates occur, the initial policy ($\pi_0$) is executed for a small number of trials ($N^*$) to collect an initial batch of real-world data.
   • Rationale: This approximates the effect of retaining simulator data when $\mathcal{D}_0$ cannot be kept, mitigating instability caused by starting with an empty online buffer.

3. Delayed Actor Updates (Asymmetric Updates):

   • Mechanism: The actor is updated less frequently than the critic (e.g., every 20 critic updates) with a conservative learning rate, while the critic is updated more frequently.
   • Rationale: This addresses the "two-timescale" problem. When dynamics shift, the critic requires time to adapt to the new distribution. If the actor updates too quickly based on a lagging critic, it reinforces errors. Delaying actor updates allows the critic to track the moving equilibrium before the policy is altered.

4. Optimized Update-to-Data (UTD) Ratio:

   • Mechanism: Scaling the number of critic updates per environment step ($\eta$) to match the scale of parallel simulation environments used for pretraining.
   • Rationale: Standard implementations often undertrain the critic when scaling to massive parallel simulators. Increasing $\eta$ improves sample efficiency and prevents undertraining, though it incurs higher computational costs.

Theoretical Analysis

The paper provides a two-timescale analysis of the "downward spiral." It models the critic as tracking an equilibrium that moves as the policy and replay distribution change.

• Equation (9) & (10): Decompose the critic tracking error into a "residual error" and a "distribution-induced drift" (or actor-induced drift).
• Equation (15): Shows that the distortion in the actor update is proportional to the tracking error and the number of critic steps ($M$) taken before the actor update.
• Conclusion: Increasing $M$ (delayed updates) reduces the finite-tracking error ($q^M$), while a smaller actor learning rate ($\beta_\pi$) limits the perturbation caused by a critic that has not yet adapted.

Key Results

The proposed recipe was validated across the three robotic platforms, demonstrating that standard off-policy RL can be made reliable without major algorithmic modifications:

• Stability vs. Instability: Baseline configurations (online-only replay, no warm start, synchronous actor-critic updates) consistently failed, leading to performance degradation or catastrophic failure (e.g., the robot falling or missing the goal). The full recipe yielded stable improvement across all tasks.
• Performance Gains:
  • Franka Emika Panda: Recovered reliable grasping capabilities that were lost in the zero-shot simulation-to-real transfer.
  • Unitree Go1: Improved robustness to previously unseen velocity commands.
  • Race Car: Achieved faster and more precise parking.
• Data Efficiency: Reusing data from previous trials (Retained Replay) significantly accelerated learning. Even retaining data from just a few preceding trials yielded substantial performance gains.
• Warm Starts: While not strictly necessary for the Franka robot, warm starts were crucial for stabilizing learning on the Unitree Go1 and Race Car.

Significance and Claims

The authors claim this work provides the first large-sample empirical study of design choices for simulation-to-online RL on physical robots.

• Practical Guidance: The paper offers a "simple recipe" built from standard RL components that practitioners can directly adopt to deploy online RL with lower engineering effort.
• Demystifying Defaults: It challenges widely used defaults (such as online-only replay or synchronous updates), showing they can be harmful in the sim-to-online setting.
• Algorithmic Minimalism: The results suggest that complex algorithmic innovations are not always necessary; rather, careful engineering of data management and optimization schedules is sufficient to make standard SAC effective for real-world fine-tuning.
• Open Source: The authors release a full-stack training pipeline and code to support ongoing research, specifically for the Franka Emika Panda.

The paper concludes by noting that while these findings make online RL more practical, fully autonomous learning (without human resets) remains an open research direction. The current work focuses on the semi-automated episodic setting where human intervention is still required for safety and resets.