Technical Summary: TC-IDM – Grounding Video Generation for Executable Zero-shot Robot Motion

1. Problem Statement

The field of embodied AI is currently divided between two paradigms, both of which face significant limitations:

• Vision-Language-Action (VLA) Models: While models like OpenVLA and RT-2 leverage large-scale datasets to transfer perceptual capabilities to robotics, their reliance on high-quality, task-specific robotic data limits their generalization to out-of-distribution (OOD) tasks and environments.
• Generative World Models: Emerging video-based world models (e.g., Sora, Kling, Cosmos) offer "visual foresight" by synthesizing high-fidelity, physically plausible future states. However, a critical "last-mile" gap exists between these pixel-level plans and physically executable actions. Raw video outputs often suffer from noisy trajectories, physical hallucinations, and kinematic inconsistencies because they lack a grounded representation of the robot's morphology and actuation limits.

Existing methods attempting to bridge this gap (e.g., AVDC, VidBot) often rely on tracking unstable object states or full pixels, making them brittle to occlusions, rapid motion, and deformable objects. The core challenge is the robust translation of high-level, pixel-based plans into low-level, physically consistent trajectories in joint-space or torque-control domains.

2. Methodology: Tool-Centric Inverse Dynamics Model (TC-IDM)

To address these limitations, the authors propose TC-IDM, a framework that anchors control policies to the stable, well-defined motion of the robot's end-effector (the "tool") as envisioned by the world model. This "plan-and-translate" paradigm operates in two distinct stages:

A. Spatiotemporal Prediction

Given an initial RGB frame, depth map, and text instruction, a generative world model produces an imagined RGB video sequence ($V_{rgb}^{gen}$).

• Depth and Pose Estimation: The system uses a foundation model (VGGT) to generate initial depth maps and relative camera poses. These are refined and metrically aligned using a completion network and least-squares alignment with a reference metric depth map, resulting in a fully metric camera trajectory and depth sequence.

B. Decoupled Action Translation

TC-IDM decomposes action translation into two complementary, decoupled streams that are concatenated to form the final control vector:

1. Vision-Driven State Generation (Gripper Control):

   • A frozen visual encoder (DINOv3) extracts temporally consistent semantic features from the generated video, capturing task cues like object identity and contact states.
   • A lightweight, task-conditioned MLP head predicts the continuous grasping aperture parameter ($A_{gripper}$) based on these semantic features.

2. Geometry-Grounded Gesture Generation (6-DoF Trajectory):

   • Trajectory Extraction: Using the metric-aligned RGB-D video and a gripper mask (generated via SAM 3), a 3D Point Tracker (SpatialTrackerv2) extracts dense 3D point trajectories of the gripper surface.
   • Rigid-Body Filtering: To ensure stability, the system enforces a rigid-body motion prior, selecting the top-$K$ trajectories that best satisfy rigid transformation consistency.
   • Pose Recovery: Instead of learning an action decoder, the system analytically recovers the 6-DoF Tool Center Point (TCP) trajectory by solving for the rigid-body transformation ($R, t \in SE(3)$) that best aligns the filtered 3D trajectories between consecutive frames. This yields the end-effector action ($A_{TCP}$).

This architecture is agnostic to the specific end-effector and can be instantiated with arbitrary tool embodiments, including dexterous hands via a retargeting module.

3. Key Contributions

• Novel Framework: The proposal of TC-IDM, which bridges world model planning and robot control by using the tool's trajectory as a robust intermediate representation, avoiding the instability of tracking deformable objects.
• Decoupled Architecture: A design that cleanly separates and exploits heterogeneous information streams: physical motion cues (for 6-DoF pose) and semantic visual features (for gripper control).
• Empirical Validation: A comprehensive set of real-world experiments establishing a new baseline for deploying generative world models on physical robots, demonstrating effectiveness across rigid and deformable objects.

4. Experimental Results

The authors evaluated TC-IDM in real-world settings across various task difficulties and video generation models.

• Overall Performance: The system achieved an average success rate of 61.11% across all tasks.
  • Simple Tasks: 77.7% success rate.
  • Zero-shot Deformable Object Tasks: 38.46% success rate.
• Comparison with Baselines: TC-IDM substantially outperformed end-to-end VLA-style baselines (e.g., $\pi_0$, RT-2) and other Inverse Dynamics Models (IDMs) like AVDC and AnyPos.
  • On "Hard" tasks requiring fine-grained 6-DoF coordination (e.g., inserting chopsticks), TC-IDM achieved significantly higher success rates than any baseline, which often failed entirely.
  • In action replay evaluations, TC-IDM achieved 93.3% on easy tasks, 53.3% on medium tasks, and 28.9% on hard tasks, setting an upper bound for task success in this evaluation.
• Generalization Capabilities:
  • Camera Variations: The model successfully transferred from Realsense D457 training data to unseen domains (Apple Pro, Realsense D435i) without fine-tuning.
  • Deformable Objects: Achieved zero-shot generalization to cloth manipulation (removal, folding) despite being trained on rigid objects.
  • Long-Horizon Tasks: Successfully executed a six-step hoodie folding sequence without intermediate re-planning.
  • Cross-Embodiment: Motion priors learned on a single-arm Franka robot were successfully transferred to a dual-arm UR robot and various dexterous hands (BrainCo, Inspire-Robots) without re-training.

5. Significance

The paper claims that TC-IDM provides a "clean and rigid structure" that is easier to learn and transfer across embodiments compared to end-to-end VLA approaches. By focusing on the tool's trajectory rather than the complex state of the manipulated object, the method overcomes the "planning-action chasm."

The significance lies in demonstrating that generative world models can be effectively grounded for physical execution. The "plan-and-translate" approach allows agents to internalize complex dynamics (from precision grasping to non-rigid folding) via visual foresight while ensuring the final output is physically consistent and executable. This establishes a scalable pathway for general-purpose embodied intelligence that does not rely on massive, task-specific robotic datasets.