Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancement

Technical Summary: Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancement

Problem Statement

Vision-Language-Action (VLA) models have demonstrated broad generalization across manipulation tasks through large-scale pretraining and imitation learning. However, policies derived from imitation learning are inherently brittle in precise physical interactions; small execution errors accumulate over time, leading to failures in unseen states. While Reinforcement Learning (RL) offers a mechanism for recovery through online interaction, directly applying RL to modern VLAs is challenging due to architectures relying on diffusion or flow matching, which are not readily differentiable through standard policy gradients.

Residual RL, which trains a lightweight corrective policy on top of a frozen base VLA, presents a natural solution. Yet, existing approaches face a fundamental "sim-to-real" dilemma:

1. Privileged-state methods require distilling a state-based teacher into an image-based student for deployment, incurring performance loss.
2. Image-based methods suffer from large visual domain gaps, preventing zero-shot transfer of sim-trained residuals.
3. Real-world RL eliminates the transfer gap but is costly, unsafe, and requires complex infrastructure.

The core challenge addressed is: Can a reinforcement learning policy trained purely in simulation improve the robustness of real-world VLAs zero-shot, without distillation or real-world RL?

Methodology

The authors propose an Object-Centric Residual RL framework that refines VLA actions using object poses, enabling a compact observation space that transfers consistently between simulation and reality. The framework operates in three stages:

1. Paired Sim/Real VLA via Teleoperation Replay
To align action distributions across domains, the authors replay the same teleoperation demonstrations used to train the real-world VLA ($\pi^{real}_{VLA}$) within a simulation environment. This trains a simulation counterpart ($\pi^{sim}_{VLA}$) alongside the real one. Since both are supervised by identical action trajectories, they learn aligned action distributions despite differing visual inputs. This allows the residual to train alongside $\pi^{sim}_{VLA}$ in simulation and transfer to $\pi^{real}_{VLA}$ zero-shot at deployment. Crucially, because the residual does not observe images, the simulation environment need not be visually realistic, reducing engineering overhead.

2. Object-Centric Residual RL Training
The residual policy ($\pi^{sim}_{res}$) is trained entirely in simulation using TD3. The key innovation lies in the observation space $s_t$, which avoids both privileged states and raw images:
$$s_t = [s^{obj}_t, s^{prop}_t, a^{base}_t]$$

• $s^{obj}_t$: 6-DoF poses (position and orientation) of task-relevant objects, estimated via off-the-shelf estimators (FoundationPose + SAM2).
• $s^{prop}_t$: Proprioceptive state (end-effector pose, gripper state).
• $a^{base}_t$: The current action chunk from the base VLA.

To ensure robustness against real-world pose estimation noise and failures, the training employs two augmentation strategies:

• Pose Noise Injection: Random perturbations are added to position and orientation (quaternion multiplication) to mimic estimator inaccuracies.
• Pose Dropout: With a certain probability, the object pose vector is zeroed out, forcing the policy to learn a fallback strategy relying solely on proprioception and the base action.

3. Zero-Shot Deployment
At deployment, both the real VLA and the sim-trained residual are frozen. The system queries the real VLA for the base action and the residual for a correction based on the observed object poses. If the pose estimator's confidence falls below a threshold, the system triggers the "dropout" fallback learned during training. The combined action is computed as $a_t = a^{base}_t \oplus \pi^{sim}_{res}(s_t)$, where $\oplus$ denotes per-component composition (addition for position/gripper, quaternion multiplication for rotation).

4. VLA Self-Improvement
Successful rollouts generated by the residual-corrected policy on the real robot can be aggregated across tasks to retrain the base VLA via supervised fine-tuning (SFT). This creates a self-improvement loop that generates higher-quality training data without requiring additional teleoperation.

Key Contributions

• Zero-Shot Sim-to-Real Framework: A novel residual RL framework that achieves zero-shot transfer without distillation or real-world RL by utilizing a domain-invariant observation space (object poses + proprioception).
• Automatic Self-Improvement: A mechanism to aggregate successful real-robot rollouts from the residual-corrected policy to retrain a single multi-task VLA, improving the base policy without new human demonstrations.
• Empirical Validation: Demonstration of consistent improvement across five manipulation tasks on a real Franka Research 3 (FR3) robot, raising the average success rate from 42% to 76%.

Experimental Results

The method was evaluated on five tabletop manipulation tasks (Cube Lift, Pick-and-Place, Stack Cube, Close Drawer, Stand Cup Up) using a real FR3 robot and a MuJoCo simulation.

• Performance Gains: The sim-trained residual improved the average success rate on the real robot from 42% (Base VLA) to 76% (Base + Residual). In simulation, the average success rate improved from 38% to 86%.
• Ablation Studies:
  • Robustness: Both pose noise injection and pose dropout were essential; removing either degraded performance, with pose dropout being critical for resilience against detection failures.
  • Observation Space: Object-centric observations significantly outperformed image-based baselines (which failed due to visual domain gaps) and distillation-based approaches (which suffered performance loss during distillation).
• Intervention Analysis: The residual consistently intervened when the base VLA was misaligned with the goal, steering actions toward the target. This selective correction resulted in 9–22% faster task completion.
• Generalization: The framework was shown to be compatible with different VLA backbones (tested on $\pi_0.5$), confirming the observation interface is not specific to a single architecture.
• Self-Improvement: Retraining the base VLA on residual-corrected rollouts yielded higher success rates and shorter episode lengths compared to retraining on original base rollouts.

Significance and Claims

The paper claims that by shifting the residual policy's observation space from images or privileged states to object-centric poses, it is possible to bridge the sim-to-real gap without complex domain adaptation or distillation. This approach leverages the generalization of VLAs while adding the precision of RL through a lightweight, zero-shot transferable module.

The authors position this as a practical path toward scalable and autonomous robot improvement. By decoupling the correction mechanism from the base policy's training and utilizing a recoverable state representation, the method enables real-world deployment of sim-trained RL policies. Furthermore, the ability to use these corrected rollouts to retrain the base VLA suggests a pathway for continuous, automated system refinement without the prohibitive cost of continuous human teleoperation.

Limitations noted by the authors:

• Reliance on real-time 6-DoF pose tracking, which can fail under full occlusion or heavy clutter.
• Requirement for manual specification of task-relevant objects.
• The pose-based observation bridges the visual gap but not the dynamics gap (e.g., friction, compliance), which may affect contact-rich tasks.
• As a residual architecture, it cannot recover from states far outside the base VLA's training distribution.
• Sub-millimeter precision tasks may exceed current pose estimation accuracy.