ObjectForesight: Predicting Future 3D Object Trajectories from Human Videos

1. Problem Definition

The paper addresses the challenge of predicting future 3D object dynamics (specifically 6-DoF trajectories) from passive, short, egocentric human videos.

• Goal: Given a sequence of RGB frames and an object mesh, the system must infer how a rigid object will move in the near future (e.g., being lifted, sliced, or rotated) based on human interaction.
• Key Challenge: Unlike standard video generation (which predicts pixels) or robot learning (which requires explicit action labels), this task requires modeling the physical consequences of human interaction in 3D space without direct supervision of the human's manipulation policy.
• Data Scarcity: A major bottleneck is the lack of large-scale, physically grounded 3D interaction datasets. Existing datasets are either small, scripted, or lack aligned 3D pose and depth information.

2. Key Contributions

The authors introduce three primary contributions:

1. Task Formalization: They define and formalize the task of 3D Object Dynamics Prediction from human videos, establishing a standardized setting for learning physical interaction priors from "in-the-wild" data.
2. ObjectForesight Model: A novel 3D object-centric dynamics model that predicts future 6-DoF trajectories. Unlike latent-space or pixel-based world models, it operates explicitly in SE(3) pose space, ensuring geometric consistency and physical plausibility.
3. Large-Scale Dataset: The creation of a dataset containing 2+ million object-centric 3D trajectories derived from the EPIC-Kitchens dataset. This was achieved via a scalable, automated pipeline that converts raw videos into metrically grounded 3D motion supervision.

3. Methodology

A. Data Curation Pipeline

To overcome the lack of 3D data, the authors developed an automated pipeline to extract high-quality 3D trajectories from EPIC-Kitchens videos:

1. Segmentation & Tracking: Uses EgoHOS for hand-object detection and SAM2 (Segment Anything Model 2) with temporal consensus to generate robust, occlusion-resilient object masks.
2. 3D Reconstruction: Uses TRELLIS to reconstruct 3D object meshes from clean views.
3. Pose Estimation & Depth: Leverages SpaTrackerV2 for metric depth and camera geometry, and FoundationPose for 6-DoF pose estimation and tracking.
4. Quality Control: Implements a two-stage filter using InternVL3 (a Vision-Language Model) to verify that objects are actually being manipulated (filtering out static scenes) and to ensure view quality.
5. Output: The pipeline yields short, clean, anchor-frame-canonicalized 6-DoF trajectories (3 input frames, 8 future frames).

B. Model Architecture

ObjectForesight combines a geometry-aware encoder with a Diffusion Transformer (DiT):

• Input Representation:
  • Context: A point cloud of the scene (from monocular depth) and a sequence of past pose tokens.
  • Pose Tokenization: Poses are reparameterized into depth-normalized 9D tokens ($u, v, s, r_{6D}$) to improve numerical stability and reduce dynamic range.
• Scene Encoder:
  • Uses PointTransformerV3 to encode the 3D point cloud.
  • Dual-Coordinate Features: Points are represented in both camera coordinates and object-centric coordinates to facilitate object-level reasoning.
  • Context Conditioning: A context vector summarizing recent motion history is injected into the encoder via FiLM (Feature-wise Linear Modulation).
  • Object-Aware Pooling: An attention mechanism pools point features into a global scene embedding ($z_{geom}$), biasing attention toward points near the object.
• Prediction Head (Diffusion Transformer):
  • A Diffusion Transformer (DiT) with AdaLN-Zero conditioning.
  • It takes the scene embedding ($z_{geom}$) and past pose tokens as conditions.
  • It predicts a distribution over future pose sequences using a v-parameterization diffusion process with a cosine noise schedule.
  • Why Diffusion? It naturally handles the one-to-many nature of future motion (e.g., a cup could be lifted or slid), allowing for multi-modal, physically plausible predictions rather than a single deterministic average.

C. Training Objectives

The model is trained with a composite loss function:

• Primary Loss: $v$-prediction MSE (standard diffusion loss) with SNR-weighted reweighting (P2 style).
• Auxiliary Losses:
  • Pose Loss: Direct supervision on SE(3) translation and rotation errors (geodesic distance).
  • Smoothness Loss: Velocity and acceleration losses to ensure temporally coherent trajectories.
  • Depth Floor Penalty: Prevents degenerate predictions where depth approaches zero.

4. Experimental Results

Datasets

• Epic-Kitchens (Curated): The primary dataset with ~2M trajectories.
• HOT3D-Clips: A lab-captured dataset with precise ground truth used for validation.

Performance Metrics

The model was evaluated on Average Displacement Error (ADE), Final Displacement Error (FDE), and their rotational counterparts (ARE, FRE).

• vs. Baselines: ObjectForesight-DiT significantly outperforms:
  • Constant Velocity: A simple linear extrapolation baseline.
  • ObjectForesight-AR: An autoregressive transformer variant (without diffusion), which struggles with multi-modal prediction.
  • Video Generation (Luma Ray3): A state-of-the-art video generator from which 3D poses were extracted. ObjectForesight achieved ~3x lower error (e.g., 0.029m ADE vs. 0.084m ADE on Epic-Kitchens), demonstrating that predicting motion directly in SE(3) is superior to inferring it from synthesized pixels.
• Ablation Studies:
  • Encoder: PointTransformerV3 (PTV3) significantly outperformed DGCNN, PointNet++, and SparseConv, highlighting the importance of geometry-aware encoders.
  • Model Scale: Increasing DiT depth (from 6L to 12L) and dimension (384D to 768D) yielded consistent improvements.
  • Context Length: 3 context frames ($C=3$) were found to be optimal; longer contexts introduced noise.

5. Significance and Conclusion

• Physical Grounding: By operating in explicit 3D pose space rather than pixel space, ObjectForesight produces trajectories that are geometrically consistent and physically plausible, making them suitable for integration into robotic manipulation systems.
• Scalability: The automated data curation pipeline demonstrates that massive, high-fidelity 3D interaction datasets can be constructed from existing internet-scale video data without manual annotation.
• Generalization: The model generalizes well to unseen objects and scenes, capturing the "affordances" of objects (how they can move) rather than just memorizing specific motion patterns.
• Future Work: The authors suggest extending the framework to articulated objects, longer prediction horizons, and direct integration with downstream planning policies.

In summary, ObjectForesight establishes a new paradigm for learning object-centric dynamics from passive observation, bridging the gap between visual perception and physical reasoning in 3D space.