Bridging the Embodiment Gap: Disentangled Cross-Embodiment Video Editing

Technical Summary: Bridging the Embodiment Gap via Disentangled Cross-Embodiment Video Editing

Problem Statement
The paper addresses the critical "embodiment gap" in learning robotic manipulation from human videos. While internet-scale human video offers a scalable solution to the data bottleneck in robotics, a substantial distribution shift exists between human demonstrators and robotic learners. This gap encompasses visual differences (appearance, viewpoint) and, more critically, morphological and dynamic differences arising from distinct kinematics, degrees of freedom, and physical affordances. Existing approaches often fail because they produce entangled representations, where task-relevant information is coupled with human-specific kinematic biases, limiting adaptability. Other methods relying on procedural pipelines (e.g., inpainting humans and overlaying robot models) often result in unrealistic video plans unsuitable for downstream policy learning.

Methodology
The authors propose a generative framework for cross-embodiment video editing that explicitly learns disentangled representations of the task and the embodiment. The core architecture consists of three main components:

1. Frozen Generative Backbone: The framework utilizes a pre-trained, frozen video diffusion model (VACE, a latent Diffusion Transformer) as the generative backbone. This leverages strong priors for high-quality, temporally coherent video synthesis without requiring full model retraining.
2. Disentangled Encoders:
   • Task Encoder ($E_{\theta_{task}}$): A trainable encoder that processes multi-modal inputs from the source human video: a text goal description ($T$), hand motion sequences ($M_s$), and object trajectories ($O_s$). It outputs an embodiment-invariant task embedding ($z_{task}$).
   • Embodiment Encoder ($E_{\theta_{emb}}$): A trainable encoder that processes a static image of the target agent's end-effector ($C_s$) to generate an embodiment-specific embedding ($z_{emb}$).
3. Parameter-Efficient Adapter: A lightweight, trainable adapter ($A_\psi$) injects the disentangled latent codes into the frozen Diffusion Transformer. This allows the model to steer generation at multiple feature levels without modifying the original backbone weights.

Training Objectives
To ensure the task and embodiment representations are truly orthogonal, the authors employ a dual contrastive learning objective alongside a standard Flow Matching (LFM) reconstruction loss:

• Disentanglement (Minimizing Mutual Information): A CLUB estimator is used to minimize the mutual information between $z_{task}$ and $z_{emb}$. This is achieved by contrasting the log-likelihood of naturally paired samples against randomly paired samples, forcing the two latent spaces to be independent.
• Intra-Space Consistency (Maximizing Similarity): An InfoNCE loss is applied separately to the task and embodiment spaces. This pulls embeddings of the same semantic class (e.g., different videos of the same task, or different images of the same robot) together while pushing different classes apart, ensuring robust and discriminative representations.
• Reconstruction: The model is trained to reconstruct the source video from noise conditioned on its own disentangled embeddings, optimizing the parameters via Flow Matching.

Key Contributions

1. Generative Disentanglement Framework: The paper introduces a novel framework that explicitly factorizes demonstration videos into orthogonal latent spaces for task and embodiment, directly addressing the distribution shift in Learning from Videos (LfV).
2. Dual Contrastive Objective: The authors propose a training strategy that simultaneously minimizes mutual information between task and embodiment spaces (ensuring independence) and maximizes intra-space consistency (ensuring stability), without requiring paired cross-embodiment data.
3. Parameter-Efficient Adaptation: By augmenting a frozen video diffusion model with trainable encoders and a lightweight adapter, the method achieves high-fidelity synthesis with minimal parameter updates.
4. Scalable Data Solution: The approach enables the synthesis of coherent, robot-specific execution videos from a single human demonstration, offering a scalable pathway to leverage internet-scale human data for robotic imitation learning.

Experimental Results
The framework was evaluated on cross-embodiment video editing tasks (e.g., grasping, pouring) using data derived from the PH2D dataset.

• Quantitative Performance: The proposed method outperformed baselines (VACE and Phantom) across video fidelity metrics (FVD, LPIPS, PSNR, SSIM). Notably, it achieved the highest scores in dynamic and temporal quality metrics (Aesthetic Quality, Temporal Style, Motion Smoothness, Overall Consistency) within the VBench suite, whereas general-purpose models struggled with dynamic plausibility.
• Qualitative Analysis: Visual comparisons showed that while baselines either hallucinated incorrect morphologies (VACE) or produced non-photorealistic overlays (Phantom), the proposed method generated videos that were both morphologically accurate and temporally coherent.
• Disentanglement Validation: t-SNE visualizations and feature correlation heatmaps confirmed that the dual contrastive objective successfully separated task and embodiment features into distinct, non-overlapping clusters.
• Downstream Validation: In a Visual Behavioral Cloning (BC) experiment using a Unitree H1 humanoid robot, policies trained on the generated robot videos achieved lower validation errors compared to policies trained on data aligned via the HAT baseline. This suggests the generated videos provide kinematic supervision better aligned with the robot's physical embodiment.

Significance and Claims
The paper claims that by learning explicitly disentangled representations, it solves the fundamental challenge of distribution shift in LfV. The method allows for the translation of human skills into "dense visual plans" for diverse robot morphologies without the need for paired human-robot data. The authors position this work as a scalable solution to the robotics data bottleneck, enabling the use of internet-scale human video to train generalist embodied agents. The significance lies in the ability to generate physically plausible, robot-centric visual demonstrations that can directly inform downstream policy learning, bridging the gap between human observation and robotic execution.