Technical Summary: Geometry-Aware 4D Video Generation for Robot Manipulation

Problem Statement

Robotic manipulation requires the ability to anticipate how the physical world changes over time, particularly regarding object motions and occlusions. While recent video generation models demonstrate strong potential for modeling dynamic scenes, they face two critical limitations when applied to robotics:

1. Temporal Coherence: Ensuring smooth, causally consistent motion over time.
2. 3D Consistency: Preserving object geometry and spatial correspondences across different viewpoints.

Existing pixel-based models (e.g., standard video diffusion) often excel at short-term motion but lack 3D structural understanding, leading to artifacts like flickering or deformation. Conversely, 3D-aware approaches often enforce geometric constraints but struggle to scale to realistic, multi-object manipulation scenarios or rely heavily on known camera poses. Furthermore, generating videos that are simultaneously temporally coherent and geometrically consistent across novel camera views remains a significant challenge.

Methodology

The authors propose a 4D video generation framework that unifies strong temporal modeling with robust 3D geometric consistency. The method generates spatio-temporally aligned future video sequences (RGB and Depth) from novel viewpoints given a single RGB-D image per view, without requiring camera poses as input during inference.

Core Architecture

The model is built upon a Latent Video Diffusion backbone (specifically adapted from Stable Video Diffusion, SVD). It jointly generates future RGB frames and 3D pointmaps.

• Input: Historical RGB-D observations from two camera views ($v_n$ and $v_m$).
• Output: Future RGB frames and 3D pointmaps for both views. Crucially, the pointmap for the second view ($v_m$) is predicted in the coordinate frame of the reference view ($v_n$).

Key Technical Components

1. Geometry-Consistent Supervision
Inspired by DUSt3R, the authors introduce a supervision mechanism to enforce cross-view alignment:

• The model predicts a pair of 3D pointmap sequences: one for the reference view ($v_n$) and one for the second view ($v_m$) projected into $v_n$'s coordinate frame.
• A Pointmap VAE (initialized from the pretrained RGB VAE and fine-tuned on pointmap data) encodes these pointmaps into latent space.
• The diffusion model is trained to minimize the difference between the reference and projected 3D points over time, learning a shared 3D scene representation.

2. Multi-View Cross-Attention
To handle the asymmetry between the native view and the projected view, the U-Net decoder architecture is modified:

• Two separate decoders (with identical architecture but independent weights) are used for the two views.
• Cross-attention layers are inserted between the decoders. Features from the reference view ($v_n$) are passed to the second view ($v_m$) decoder via cross-attention. This allows the second view to incorporate geometric cues from the reference view, facilitating accurate pointmap prediction in the shared coordinate frame.

3. Joint Optimization
The training objective combines the standard video diffusion loss (for RGB) with the proposed geometry-consistent loss (for pointmaps):
$$L = \sum_{t'} \left[ (L^n_{diff}(t') + L^m_{diff}(t')) + \lambda \cdot L^{3D-diff}(t') \right]$$
Where $L^{3D-diff}$ applies diffusion losses to both the native view pointmaps and the projected view pointmaps. A weighting mechanism is also applied to prioritize accuracy in the robot gripper region, which is critical for downstream action extraction.

4. Robot Policy Extraction
The generated 4D videos are not just for visualization; they are used to recover robot end-effector trajectories:

• An off-the-shelf 6DoF pose tracker (FoundationPose) processes the generated RGB-D sequences to extract the gripper pose ($T_t \in SE(3)$).
• Gripper open/close states are inferred by segmenting fingers in the predicted pointmaps and measuring the distance between centroids.
• This allows the system to bypass the need for explicit camera pose calibration during inference, as the model predicts geometry directly in the reference frame.

Key Contributions

1. Unified 4D Generation Framework: A novel model that simultaneously achieves temporal coherence and multi-view 3D geometric consistency, addressing the gap between pixel-based video generation and 3D-aware reconstruction.
2. Geometry-Consistent Supervision: The introduction of a cross-view pointmap alignment loss during training, enabling the model to learn shared 3D scene representations without requiring camera poses at inference.
3. Multi-View Cross-Attention Mechanism: A specific architectural adaptation in the diffusion decoder to enforce 3D alignment between views, proven essential for generating consistent pointmaps.
4. Robot Manipulation Benchmark: The establishment of a benchmark comprising both simulated (LBM/Drake) and real-world robotic tasks with diverse camera viewpoints to evaluate 4D generation quality and generalization.
5. Downstream Application: Demonstration that the generated 4D videos can directly drive robot manipulation policies via pose tracking, achieving high success rates on tasks with unseen camera viewpoints.

Experimental Results

The method was evaluated on three simulated tasks (StoreCerealBoxUnderShelf, PutSpatulaOnTable, PlaceAppleFromBowlIntoBin) and four real-world tasks.

• Video Quality: The proposed method outperforms baselines (including SVD, SVD with multi-view attention, and 4D Gaussian reconstruction) in Fréchet Video Distance (FVD) for both reference and novel views.
• Depth and Geometry: The method achieves superior depth prediction accuracy (lower AbsRel, higher $\delta_1$) and significantly higher cross-view consistency (measured by mIoU of gripper masks) compared to baselines.
• Ablation Study: Removing the multi-view cross-attention mechanism ("OURS w/o MV attn") resulted in a significant drop in 3D consistency (mIoU) and depth accuracy, confirming the necessity of the geometric supervision and attention mechanism.
• Robot Policy Performance: When used to extract trajectories for robot control, the proposed method achieved an average task success rate of 64% across three simulation tasks. This significantly outperformed baselines like Dreamitate (12%), Diffusion Policy (12%), and DP3 (25%), demonstrating that geometric consistency is critical for successful visuomotor control.

Significance and Claims

The paper claims that by enforcing geometric consistency through cross-view pointmap alignment, the model learns a shared 3D representation that generalizes well to novel viewpoints. This is particularly significant for robotics, where small camera shifts can cause visuomotor policies to fail.

The authors emphasize that their approach enables:

• Pose-Agnostic Inference: The ability to generate consistent 4D videos from novel views without explicit camera pose inputs.
• Direct Action Extraction: The capability to use generated videos directly with standard pose trackers to derive robot actions, bridging the gap between generative world models and physical control.
• Robustness: Improved performance in complex, multi-object manipulation scenarios where occlusions and dynamic interactions are present.

The work concludes that while current inference speeds are slower than end-to-end behavior cloning, the trade-off yields significantly higher fidelity and geometric consistency, which are prerequisites for reliable robot manipulation in unstructured environments. Future work may explore faster generative models (e.g., flow matching, autoregressive transformers) to enable closed-loop planning.