Robot-Factored World Models via Robot Rendering

Technical Summary: Robot-Factored World Models via Robot Rendering

Problem Statement
Action-conditioned video world models aim to predict future observations from an initial state and an action signal, enabling planning and policy improvement without real-world interaction. However, a fundamental challenge exists in how robot actions are presented to these models. In physical robotics, actions influence future observations through two distinct processes: (1) Action Realization, where control signals are converted into robot motion by the robot's specific body, controller, and kinematics; and (2) Scene Response, where the environment reacts to that motion via contact and object dynamics.

Existing approaches condition directly on robot-specific control signals (e.g., joint torques or Cartesian poses), forcing the world model to learn the realization process itself, which is embodiment-specific and redundant. Conversely, conditioning on logged future states (realized trajectories) leaks the interaction outcomes (contact, compliance, latency) that the model is intended to predict. The core problem is identifying a visual conditioning signal that is available at deployment, avoids learning the realization process, and does not leak future interaction outcomes.

Methodology
The authors propose Robot-Factored World Models, which decouple the robot-specific realization and rendering processes from the world model itself. The methodology introduces a shared visual interface composed of three main components:

1. Nominal Trajectory Conditioning (Action Realization):
   Instead of feeding raw control commands or realized states, the system processes the action sequence $a_{1:F}$ through the robot's own controller and kinematics in a scene-free environment to generate a nominal trajectory $q_{1:F}$. This trajectory represents the motion the robot would execute in free space before any scene interaction occurs. It serves as a deployment-available signal that bridges the gap between raw commands and realized states without encoding contact outcomes.

2. Robot Rendering (Visual Interface):
   The nominal trajectory is rendered into the target camera frame using the robot's URDF (Unified Robot Description Format). This produces:

   • Robot Mesh RGB ($M^{rgb}_{1:F}$): A video of the robot's geometry (links, gripper, end-effector) as it would appear in the scene.
   • End-Effector Depth ($D^{eef}_{1:F}$): Depth information specifically for the end-effector.
     This rendering factors out the robot's geometry, kinematics, and appearance, presenting the action to the world model purely as visible robot geometry in pixel space.

3. Camera-Aware Static Context:
   To provide scene context without leaking future dynamics, the initial scene state $S_0$ is rendered along the same camera trajectory $C_{1:F}$ to produce static scene RGB ($B^{rgb}_{1:F}$) and scene depth ($D^{scene}_{1:F}$). This stream provides appearance and geometric context independent of the robot's interaction.

Model Architecture
The world model is instantiated as a latent video diffusion model (specifically using a video inpainting backbone). The conditioning input $c$ is formed by concatenating the encoded latent representations of the static scene context, rendered robot mesh, scene depth, and end-effector depth. The model learns to predict the future video $V_{1:F}$ given these visual inputs and a text prompt $T$ (which describes the scene context but excludes the action or future outcome). The training objective minimizes the flow-matching loss between the predicted velocity and the target velocity in the latent space.

Key Contributions

1. Deployment-Realistic Interface: The authors introduce a visual interface that realizes actions into nominal trajectories and renders them as camera-aligned URDF mesh RGB and end-effector depth. This factors out scene appearance and viewpoint into a static context stream.
2. Depth-Awareness: The interface is augmented with end-effector and scene depth to resolve geometric ambiguities (proximity, occlusion, and likely contact) that RGB image-plane overlap alone cannot distinguish.
3. Embodiment and Source Generalization: By representing actions as rendered geometry rather than embodiment-specific control signals, the interface decouples the world model from specific robot controllers. This enables generalization to unseen robot embodiments and allows for grounded robot-interaction video generation from retargeted human demonstrations.

Experimental Results
The method was evaluated on the DROID and RoboCasa-GR1 datasets, comparing the rendered interface against vector-conditioned baselines (Ctrl-World with Cartesian pose conditioning and an AdaLN state-vector baseline).

• Quantitative Performance: The rendered interface outperformed vector-conditioned baselines across PSNR, SSIM, and LPIPS metrics on both datasets. Specifically, the combination of nominal mesh rendering and depth cues achieved the best results (e.g., 24.61 PSNR on RoboCasa-GR1 vs. 17.67 for the state-vector baseline).
• Qualitative Analysis: The rendered interface better localized scene changes around the robot's motion and contact regions. Ablation studies confirmed that using nominal trajectories (rather than raw actions) improved alignment, and adding depth cues significantly helped the model distinguish between image-plane overlap and actual physical contact.
• Generalization: The model successfully generalized to unseen robot embodiments (e.g., an xArm 6 with an Inspire F1 hand) by simply rendering the new geometry into the same interface. Furthermore, it successfully generated robot manipulation videos from retargeted human hand motions, demonstrating that the interface is agnostic to the motion source.

Significance and Claims
The paper claims that by factoring out action realization and robot rendering, the world model is freed from learning embodiment-specific control dynamics and can focus entirely on the shared problem of predicting how a scene responds to visible robot geometry. This approach resolves the trade-off between action-realization learning and future-state leakage. The authors assert that this "robot-factored" approach provides a consistent visual interface across different viewpoints and robot embodiments, enabling more robust planning and the generation of grounded interaction videos from diverse motion sources, including human demonstrations.

Limitations
The authors acknowledge that the approach requires a known robot URDF and camera-to-robot calibration. Additionally, the static context stream assumes a static scene representation, which is fully available in simulation but only partially observable in the real world without feedforward 3D reconstruction. Finally, the training data (DROID) is predominantly successful, meaning the model has limited exposure to grasp failures or slips, which may affect robustness in those specific scenarios.