Technical Summary: VE2VF – Vision-Enabled to Vision-Free Distillation

Problem Statement

Robotic assembly and contact-rich manipulation require high precision and adaptability to diverse parts and environments. Traditional model-based approaches struggle due to the difficulty of accurately modeling contact dynamics, friction, and deformation. While learning-based approaches like Reinforcement Learning (RL) offer promise, they face significant hurdles:

• Sim-to-Real Gap: Simulators often fail to capture complex contact interactions, making policy transfer to the real world difficult.
• Sample Inefficiency: Direct model-free RL on hardware requires excessive interaction time and safe exploration strategies.
• Visual Overfitting: Vision-enabled policies, while effective during training, tend to overfit to specific visual conditions (lighting, textures, backgrounds). This limits their robustness and generalization to out-of-distribution (OOD) tasks or unseen environments.
• Partial Observability: Training vision-free policies directly from scratch using only proprioception (pose, twist, wrench) is challenging due to the lack of visual guidance for exploration.

Methodology: VE2VF Framework

The authors propose VE2VF (Vision-Enabled to Vision-Free), a human-in-the-loop (HIL) RL framework that employs a teacher-student distillation approach to achieve robust, vision-free manipulation. The process consists of three main stages:

1. Human-in-the-Loop RL Setup

The system utilizes Soft Actor-Critic (SAC) with RLPD (Reinforcement Learning with Prior Data). The learning loop integrates:

• Autonomous Exploration: The agent generates its own transitions.
• Human Guidance: A human operator provides offline demonstrations and online teleoperation interventions (overwriting agent actions) to guide exploration and provide reward signals.
• Reward Structure: A sparse binary reward is used (1 for task success, 0 otherwise), avoiding complex reward shaping.

2. Stage I: Vision-Enabled Teacher Training

A "teacher" policy ($\pi_t$) is trained on a set of representative tasks using full multimodal inputs:

• Observations: Images from cameras ($I$), end-effector pose ($p$), velocity ($v$), force ($f$), and torque ($\tau$).
• Representation: The pose is represented relative to the initial state of the episode to facilitate reactive visual servoing.
• Goal: The teacher learns to leverage rich visual feedback to master the tasks, achieving high success rates. However, it remains dependent on visual conditions.

3. Stage II: Vision-Free Student Distillation

A "student" policy ($\pi_s$) is trained to replicate the teacher's behavior but relies solely on proprioceptive inputs (pose, twist, wrench), discarding visual data.

• Distillation Mechanism: The student is trained using the same HIL-RL framework but with additional loss terms to align with the teacher:
  • Critic Distillation: A Mean Squared Error (MSE) term forces the student's Q-function ($Q_s$) to match the teacher's Q-function ($Q_t$).
  • Actor Distillation: A Kullback-Leibler (KL) divergence term regularizes the student's policy ($\pi_s$) to match the teacher's action distribution ($\pi_t$).
• State Representation: The student uses a task-relative coordinate system ($T_z$) to ensure invariance to absolute spatial positioning.
• Outcome: The student inherits the "skills" and exploration strategies learned by the teacher but becomes invariant to visual shifts.

4. Stage III: Fine-Tuning (Optional)

For highly challenging or unseen tasks, the vision-free student can be fine-tuned. This involves training a new vision-enabled teacher specialized for the specific task and distilling its knowledge into the student, rather than training the student from scratch.

Key Contributions

1. Cross-Modal Distillation Framework: A novel method that distills knowledge from a vision-enabled teacher to a vision-free student. This retains the sample efficiency of visual learning during the training phase while producing a deployment policy that is invariant to visual conditions.
2. Efficient Skill Generalization: The approach demonstrates that co-training on a small set of representative tasks enables robust zero-shot transfer to unseen insertion scenarios.
3. Real-World Validation: The method is validated entirely on real-world hardware (Franka FR3) without domain randomization or data augmentation, achieving high success rates in approximately 50 minutes of total robot interaction time.

Experimental Results

Experiments were conducted on the NIST Assembly Board I benchmark, involving various peg, gear, and connector insertion tasks.

• Performance: The VE2VF method achieved a 95% overall success rate across training, disturbed, and out-of-distribution tasks.
• Comparison to Baselines:
  • Vision-Enabled RL (HIL-SERL VPTW): Achieved near-perfect success on training tasks but collapsed (0% success) on out-of-distribution tasks and degraded significantly under visual disturbances.
  • Vision-Free RL from Scratch (HIL-SERL PTW): Showed moderate performance (47.1%) but struggled to learn effective insertion strategies without visual guidance, failing to reach full success.
  • Dynamic Motion Primitives (DMP): Performed poorly (35.7%), confirming the need for adaptive, learned behaviors.
  • Residual RL: Performed well (85.7%) but was outperformed by VE2VF, particularly in recovery behaviors (e.g., slipping off a socket and retrying).
• Ablation Studies:
  • Distillation was proven essential; training the vision-free student from scratch, even with extended time (up to 75 minutes), led to overfitting on simpler tasks and failure on complex ones.
  • Combining pose, twist, and wrench inputs provided the best robustness compared to using subsets of these modalities.
• Fine-Tuning: For the challenging DSUB insertion task (50% zero-shot success), fine-tuning the student with a task-specific teacher via distillation achieved 100% success in just 10 additional minutes.

Significance and Claims

The paper claims that VE2VF addresses the critical trade-off between the sample efficiency of vision-guided learning and the robustness required for real-world deployment. By using vision only as a "teacher" to guide exploration and then distilling that knowledge into a proprioceptive "student," the system achieves:

• Robustness: Policies that generalize across different lighting, textures, and target poses.
• Adaptability: The ability to recover from failures (e.g., slipping) through reactive behaviors learned from the teacher.
• Efficiency: Rapid acquisition of complex contact-rich skills (approx. 50 minutes) without the need for extensive simulation or manual reward shaping.

The authors conclude that this approach offers a practical pathway for deploying robust contact-rich manipulation skills in high-mix, low-volume assembly settings where visual conditions are unpredictable.