Technical Summary: Structured 4D Latent Predictive Model for Robot Planning

Problem Statement

Current robotics planning paradigms face significant limitations in generalization and spatial reasoning. While end-to-end policy-based agents (mapping observations directly to actions) perform well in narrow, instrumented settings, they struggle with distribution shifts such as changes in lighting, viewpoint, or task composition. Conversely, video-based predictive models, which have emerged as a promising alternative for long-horizon planning, operate inherently in 2D pixel space. This 2D nature leads to physical inconsistencies with the real 3D world, resulting in a lack of accurate spatial understanding and multi-view coherence. Existing attempts to incorporate 3D cues (e.g., predicting depth and normals alongside RGB) often remain surface-level projections, vulnerable to occlusions and viewpoint shifts. Traditional 3D representations (point clouds, meshes) lack the rich visual detail required for semantic understanding, while photorealistic representations (NeRFs, 3D Gaussian Splatting) are computationally intensive and difficult to model dynamically. The core challenge addressed is building a model that inherently simulates the dynamic 3D structures of the world to enable precise, 3D-consistent robot planning.

Methodology

1. Structured 3D Latent Representation

The authors propose a Structured 4D Latent Predictive Model that operates in a structured latent space rather than pixel space.

• Representation: The scene is encoded into a sequence of sparse voxel grids. Each active voxel $p_i$ holds a compact feature vector $f_i$ encoding local geometry and color. This balances structural information with latent compression, using approximately 8,000 active voxels with 8-dimensional features, compared to dense grids requiring $64^3$ elements.
• Encoding: Multi-view RGB-D observations are unprojected using camera geometry to form a 3D point cloud, which is voxelized. Pre-trained DINOv2 encoders extract patch-level embeddings from images, which are unprojected and averaged into the voxel grid to produce the structured 3D latents $z_t$.
• Decoding: A latent decoder maps voxel features to sets of 3D Gaussians. These Gaussians can be rendered into images from arbitrary viewpoints or converted into explicit point clouds, bridging raw visual perception and the internal 3D state.

2. Predictive Model Architecture

The model predicts the evolution of the 3D scene ($z_{t+1}, \dots, z_{t+T}$) conditioned on the current state $z_t$ and a textual instruction $c$. It employs a two-step pipeline:

• Single Dynamics Model (SD): Focuses on coarse geometric changes. It predicts the sparse voxel positions of the next state using conditional flow matching with a transformer backbone. It takes the current latent and text instruction as input.
• Latent Generator (LG): Focuses on appearance and visual details. Given the voxel positions predicted by the SD, the LG fills in the feature vectors for those voxels, also using a flow-matching framework.
• Training: Both components are trained independently as conditional flow-matching models. The training objective minimizes the distance between the predicted velocity field and the true noise vector in the interpolation between clean data and Gaussian noise.

3. Planning Pipeline

The predictive model serves as a high-level planner, generating future 3D states that are translated into executable actions via a Goal-Conditioned Inverse Dynamics Module:

• Latent to Action: The model predicts a sequence of latent subgoals. An inverse dynamics module maps the current state and the next subgoal to a short-horizon action chunk (joint positions).
• Implementation Options:
  1. Learned Inverse Dynamics: Decodes latents to point clouds, encodes them with a pyramid convolutional backbone, and uses a diffusion head to predict actions.
  2. Learning-Free Pose Estimation: Decodes the predicted subgoal to a point cloud, estimates the rigid transformation aligning the robot gripper template to the predicted gripper geometry (using FPFH, RANSAC, and ICP), and passes the target pose to a motion planner.
• Execution: The system can operate in open-loop (executing predicted actions) or closed-loop (replanning after each step based on new observations).

Key Contributions

1. Structured 4D Latent Predictive Model: A novel model that predicts future 3D structures conditioned on observations and text, achieving strong visual quality, 3D consistency, and robust viewpoint generalization by operating in a structured latent space rather than 2D pixel space.
2. Planning Framework: A framework that leverages detailed 3D predictions as geometrically rich goals for an inverse dynamics module, enabling precise and spatially-aware manipulation.
3. Empirical Superiority: Demonstrated performance outperforming state-of-the-art video-based planners (e.g., UniPi, TesserAct) in both generation quality and downstream robotic task performance, including strong zero-shot generalization to visual changes and effectiveness on real-world platforms.

Experimental Results

4D Generation Quality

• Metrics: Evaluated on image quality (PSNR, SSIM, LPIPS) and 3D consistency (Chamfer Distance, depth error, multi-view PSNR/SSIM/LPIPS).
• Performance: The proposed method achieved superior 3D consistency and multi-view coherence compared to baselines like Wan-2.1, TesserAct, and OpenSora-1.3. For instance, it achieved a Chamfer Distance of 5.95 compared to ~42-44 for baselines, and a multi-view cPSNR of 27.42 vs. ~16-17 for baselines.
• Visual Fidelity: Generated sequences maintained temporal consistency and high visual fidelity across multiple camera views.

Robot Planning Performance

• Simulation (ManiSkill3 & RLBench): On tasks like StackCube, ToolPull, and PegInsertion, the method achieved an average success rate of 61.3% on ManiSkill3, significantly outperforming UniPi (29.3%) and TesserAct (24.7%), and approaching specialized imitation learning policies. On RLBench tasks, it achieved a 75.3% average success rate compared to 55.0% (UniPi) and 61.7% (TesserAct).
• Zero-Shot Generalization: The model demonstrated robustness to unseen visual conditions (lighting changes, noise, background shifts) and novel viewpoints (up to 10° rotation), maintaining success rates above 78% where baselines dropped significantly (e.g., DP dropped to 7% under lighting changes).
• Real-World Validation: In a real-world block-in-basket task with 50 episodes, the method achieved an 80% success rate, outperforming a Diffusion Policy baseline (62%).

Ablation Studies

• Input Representation: Decoded point clouds were found to be the most effective input for the inverse dynamics module, offering a balance of geometric fidelity and computational efficiency compared to raw voxels or full 3D latents.
• Viewpoint Count: While training with more views (40) improved 3D consistency, the model remained effective with limited training views (4), especially when using closed-loop replanning, which substantially boosted success rates.

Significance and Claims

The paper claims that by moving beyond 2D video-based approaches to a structured 4D latent space, the proposed model overcomes the inherent physical inconsistencies and lack of spatial reasoning found in current video generative planners. The significance lies in:

• Holistic 3D Understanding: The ability to encode and predict the complete 3D scene structure, ensuring multi-view coherence and physical consistency.
• Generalization: The capacity to generalize to novel visual conditions and viewpoints without retraining, a critical requirement for real-world deployment.
• Modular Planning: The separation of the predictive model (planning) from the inverse dynamics module (control) allows for flexible integration and transfer across different robot setups and tasks.
• Real-World Applicability: The successful transfer of the method from simulation to real-world robotic platforms demonstrates its potential for practical deployment in complex manipulation tasks.

The authors note that while the current implementation assumes calibrated multi-view inputs for initial reconstruction, extending the approach to weaker input settings remains a direction for future work.