Human Preference Modeling Using Visual Motion Prediction Improves Robot Skill Learning from Egocentric Human Video

1. Problem Statement

The paper addresses the challenge of transferring skills from large-scale, unstructured egocentric human video datasets (e.g., Ego4D, Epic Kitchens) to robots.

• The "Skill Gap": Robots trained solely on limited on-robot teleoperation demonstrations often underperform compared to human demonstrations due to the high cost of collecting robot data and the limited coverage of the task space.
• Limitations of Existing Methods: Prior approaches to learning rewards from human videos typically estimate the long-term value of a state based on its temporal distance to the goal (i.e., how many frames remain until the end of the video).
  • Flaws: This approach is susceptible to noise (e.g., human hesitation, multitasking, or non-optimal pacing), which negatively biases the value of all preceding states.
  • Distribution Shift: These methods struggle to transfer learned value functions across the "embodiment gap" (differences between human and robot kinematics/environment) because they rely on global temporal metrics rather than local physical consistency.
• Goal: Develop a method to learn a robust, dense reward signal from human videos that allows a robot to improve its policy via Reinforcement Learning (RL) with minimal on-robot data.

2. Methodology

The authors propose a framework called Motion Prediction Reward (MPR). Instead of estimating long-term value, the method models short-term human preferences by predicting the motion of task-relevant objects.

A. Core Concept: Motion Prediction as Preference

The system assumes that human demonstrators implicitly define a "correct" motion for objects at every step.

1. Point Tracking: The system tracks a grid of points on task-relevant objects (and background) across human video frames.
2. Motion Prediction Model ($F_\theta$): A neural network (based on a modified Diffusion Transformer architecture with DinoV2 visual features) is trained to predict the location of these points in the next frame ($t+1$) given the current observation ($o_t$), current point locations ($P_t$), and normalized task progress ($t/T$).
   • Training Data: Large-scale egocentric human videos.
   • Loss Function: Regression loss between predicted and actual point locations.

B. Reward Signal Generation

For a robot attempting the task, the reward is calculated at each time step based on the alignment between predicted motion (from the human model) and observed motion (from the robot's video).

• Formula: The reward $r_t$ is the sum of the positive cosine similarities between the predicted point deltas ($\Delta P^{pred}$) and the tracked point deltas ($\Delta P^{track}$) for all points on task-relevant objects.
  $$r_t = \sum \max\left( \frac{\Delta P^{pred}_t \cdot \Delta P^{track}_t}{\|\Delta P^{pred}_t\| \|\Delta P^{track}_t\|}, 0 \right)$$
• Advantage: This rewards the robot for achieving the outcome (object motion) rather than mimicking the specific human action, making it robust to embodiment differences. It also avoids bias from human hesitation, as only the current step's motion is evaluated.

C. Learning Framework: Residual Reinforcement Learning

To optimize the robot policy using this reward:

1. Base Policy: A behavior cloning policy ($\pi_{base}$) is trained on a small set of on-robot demonstrations (10 episodes).
2. Residual RL: A modified Soft Actor-Critic (SAC) algorithm learns a residual policy ($\pi_{RL}$) that outputs corrective actions ($\Delta a$).
   • Final Policy: $\pi(s) = \pi_{base}(s) + \alpha \pi_{RL}(s)$.
   • Stability Mechanisms: The base policy is frozen. The critic network is pretrained on the offline data and uses techniques (LayerNorm, min-Q) to prevent value overestimation. The system samples equally from offline demonstrations and online interaction buffers.

3. Key Contributions

1. Novel Reward Formulation: Shifts from temporal-distance-based value estimation to visual motion prediction. This explicitly encodes human preferences as "how objects should move," filtering out noise like hesitation.
2. Robustness to Distribution Shift: By focusing on the motion of task-relevant objects rather than global state similarity, the method bridges the gap between human and robot videos more effectively.
3. Sample Efficiency: Demonstrates that a robot can significantly improve performance using only 10 on-robot demonstrations and ~1 hour of real-world interaction (100 episodes).
4. Failure Discrimination: The motion-based reward naturally penalizes failed attempts where the robot moves but the object does not (e.g., missing a handle), whereas temporal-distance methods often assign positive rewards to these "empty motions."

4. Experimental Results

The authors evaluated the method on three tasks: "Open Microwave," "Fold Cloth," and "Wipe Counter."

• Simulation (Franka Kitchen Benchmark):

  • MPR outperformed VIP (Value Implicit Pretraining, a state-of-the-art temporal-distance method) and matched a hand-crafted dense reward function.
  • VIP showed bias and local minima issues, particularly in the "Open Microwave" task.

• Real-World Hardware (Franka Research 3):

  • Performance Gains:
    • Open Microwave: MPR improved success rate from 45% (base) to 76.7%. VIP dropped to 23.3% (unlearning).
    • Fold Cloth: MPR improved from 45% to 73.3%. VIP dropped to 8.3%.
    • Wipe Counter: MPR improved from 40% to 56.7%.
  • Stability: VIP runs exhibited "forgetting" behavior, where the robot learned to maximize value by performing motions that looked like the task but failed to achieve the goal (e.g., moving the end-effector to the goal position without grasping the cloth). MPR correctly identified these as failures because the object did not move.
  • Efficiency: MPR achieved these results with only ~15,000 steps (100 episodes), whereas VIP required significantly more data to converge (if at all).

5. Significance and Conclusion

This work presents a paradigm shift in learning from human video. By modeling short-term physical preferences (object motion) rather than long-term temporal proximity, the authors created a reward signal that is:

• Dense and Informative: Provides feedback at every step.
• Embodiment-Agnostic: Focuses on the physics of the task rather than the specific kinematics of the actor.
• Robust: Resistant to noise in human demonstrations (hesitation, multitasking).

The approach enables robots to autonomously refine skills in the real world with minimal human intervention, offering a scalable path toward general robot learning from the vast amounts of video data available on the internet. The authors suggest future work could extend this to language-conditioned multi-task models and explore world-model-based search to further optimize policies.