💻 computer science

MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction

MotionForesight is a method that re-purposes pretrained video prediction models to forecast future 3D scene flow for human-object interactions by training a lightweight adapter on pseudo-ground-truth tracks, enabling efficient generalization across diverse objects and environments without auxiliary inputs.

Original authors: Homanga Bharadhwaj, Yash Jangir

Published 2026-07-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Homanga Bharadhwaj, Yash Jangir

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a magician's assistant reach for a floating deck of cards. You don't need to see the cards fly into the air to know what happens next; your brain instantly simulates the physics. You know the cards will lift, perhaps fan out, and maybe flutter down. This ability to predict the future based on the present is a superpower humans have, but it's incredibly hard to teach to robots. In the world of computer science, specifically a field called "embodied intelligence," researchers are trying to build machines that can do the same thing: look at a video of a human interacting with an object and guess exactly how that object will move in 3D space.

To understand the challenge, think of the difference between watching a movie and reading a map. Most AI models that predict the future are like movie directors; they try to generate the next few seconds of a video, pixel by pixel, to see what the scene looks like. But a robot doesn't care about the color of the shirt or the texture of the wall; it cares about the geometry—the precise 3D path a cup will take as it slides off a table. This paper tackles the problem of teaching AI to skip the "movie generation" step and go straight to drawing the "map" of future motion, using only a short clip of a human doing something everyday, like opening a drawer or lifting a box.

The researchers behind this project, from Johns Hopkins University, have developed a system they call MotionForesight. Their big idea is surprisingly simple: instead of training a robot from scratch to understand physics, why not borrow the "common sense" that already exists in massive video models? They took a powerful AI that was already trained to track how objects move in existing videos (a "retrospective" tracker) and tricked it into becoming a "proactive" predictor.

Here is how they pulled off this magic trick. Imagine you have a very smart friend who is great at looking at a finished puzzle and telling you where every piece went. That's what the original video model does; it looks at a whole video and tracks points on an object. The researchers asked, "What if we show this friend only the first half of the puzzle and ask them to guess where the pieces will go in the second half?" To do this, they took 40,000 videos of humans interacting with objects (mostly from a dataset called Something-Something V2) and used the AI to generate "fake" perfect tracks of what happened in the full videos. Then, they trained their new model, MotionForesight, using only the first part of those videos, forcing it to guess the rest.

The result is a system that can look at a short clip—say, a hand reaching for a mug—and predict the exact 3D path the mug will take for the next 15 steps (about 0.5 seconds of future time), all without needing to know what the object is called or what the human is trying to say. It doesn't need language instructions like "pick up the cup"; it just watches the motion and figures out the physics.

The paper suggests that this approach is incredibly efficient. While other methods try to learn from millions of videos or require complex language descriptions to work, MotionForesight achieved better results using just 40,000 videos and no language at all. When tested on new, unseen videos taken with phone cameras in different homes and offices, the model successfully predicted motions like lifting, sliding, and rotating objects. It even outperformed much larger models that were trained on over a million videos and given extra help with language labels.

The authors found that by repurposing the "memory" of a video model that already understands how things move, they could create a lightweight tool that gives robots a clear, geometric roadmap of the future. They showed that the model can handle tricky situations, like a drawer sliding on a track or a lid rotating shut, and it works even on objects it has never seen before. The paper concludes that this method is a promising step toward giving robots the kind of intuitive "gut feeling" about motion that humans have, allowing them to anticipate the physical consequences of their actions before they even happen.

Technical Summary: MotionForesight

Problem Definition

The paper addresses the challenge of future 3D scene-flow prediction for manipulated objects from casual, monocular human-object interaction videos. Unlike traditional approaches that predict future rigid 6-DoF poses (which fail for articulated or deformable objects) or future 2D image tracks (which lack metric depth), MotionForesight aims to predict dense, reference-anchored 3D trajectories for points on an object.

The core difficulty lies in learning this anticipation from "passive" observation without explicit language instructions, action labels, or knowledge of the object's physical properties. The goal is to infer the physical consequences of an interaction (e.g., how a cup lifts or a drawer slides) based solely on a short observed video prefix.

Methodology

Core Insight

The authors propose that modern video models, trained on large corpora, already encode rich priors about temporal regularities, contact, affordances, and plausible dynamics. Instead of training a model to generate future RGB pixels (which is computationally expensive and often loses geometric fidelity), MotionForesight redirects these video priors toward predicting explicit future 3D motion fields.

Architecture and Adaptation

MotionForesight is built upon TrackCraft3R [11], a dense 3D tracker constructed on top of a pretrained video Diffusion Transformer (DiT) (specifically Wan2.1 [12]). The method repurposes this retrospective tracker into a forward predictor through the following modifications:

Input Representation:
- Observed Context: For the first $T_1$ frames, the model receives RGB frames and estimated pointmaps (derived from monocular depth estimation).
- Future Context: For the future $T_2$ steps, actual RGB and pointmap data are unavailable. These slots are replaced by learned mask latents ( $m_{rgb}, m_{pm}$ ). These masks indicate "unobserved" time steps without encoding specific future appearances.
- Query Stream: A reference-frame query latent (derived from the object mask in the first frame) is repeated across all timestamps ( $T_1 + T_2$ ), asking the model: "Where is this reference point at time $t$ ?"
Training Strategy:
- Pseudo-Ground Truth: The authors curate a dataset of ~40K human-object interaction videos (from Something-Something V2). They run the full TrackCraft3R tracker on complete clips offline to generate dense 3D trajectories, which serve as pseudo-ground-truth targets.
- Masking during Training: During training, the model is fed only the observed prefix ( $T_1$ ) and the masked future slots. It must predict the residual track latents for the future steps.
- Parameter Efficiency: The large video backbone (DiT), the original tracking adapter, and the VAE encoders/decoders are frozen. Only a lightweight, low-rank adapter (LoRA), prediction head, and the mask latents are trained.
Objective Function:
The model is trained using a decoded coordinate-space loss ( $L_{dec}$ ) that minimizes the Euclidean distance between predicted and pseudo-ground-truth 3D coordinates. The loss is weighted to prioritize future prediction ( $\lambda_{fut}$ ) over observed frame reconstruction ( $\lambda_{obs}$ ), ensuring the model learns to extrapolate motion rather than just memorize the past.

Key Contributions

Formalization of the Task: The paper formalizes future 3D scene-flow prediction from casual RGB videos as a reference-anchored tracking-pointmap forecasting task, abstracting away from specific object categories or rigid-body assumptions.
Data Curation Pipeline: A novel pipeline that converts unstructured human videos into pseudo-ground-truth 3D trajectories using dense 3D tracking and object masks, enabling supervision without metric 3D sensors.
Efficient Model Repurposing: A minimal modification to TrackCraft3R that transforms a retrospective tracker into a forward predictor. This approach demonstrates that freezing the heavy video and tracking components while training a small adapter is sufficient to learn geometric forecasting.

Experimental Results

The model was trained on 40,000 human-object interaction videos from SSv2 and evaluated on held-out SSv2 clips and Out-of-Distribution (OOD) phone videos containing unseen objects and environments.

Performance vs. Baselines: MotionForesight outperforms significantly larger models, including MolmoMotion (trained on 1.16M videos) and video generation pipelines followed by tracking.
- On OOD phone videos, MotionForesight achieved an Average Displacement Error (ADE) of 9.31 cm, compared to 9.50 cm for MolmoMotion (without language) and 13.82 cm for video generation baselines.
- Crucially, MotionForesight achieves this without any language or action labels, whereas MolmoMotion relies on language-conditioned inputs.
Scaling: Experiments showed that increasing the training data from 1K to 40K videos improved the model's ability to generalize to diverse, unseen scenarios, particularly in terms of the Percentage of Predictions within a Threshold (PWT).
Qualitative Results: The model successfully predicts diverse motions including lifting, rotation, constrained sliding, and non-rigid deformations, maintaining spatial coherence even in long-horizon predictions.

Significance and Claims

The paper claims that efficiently re-purposing video priors is a viable path to embodied intelligence. By bypassing the generation of future pixels and directly predicting metric 3D motion, the method provides an actionable representation with lower computational overhead.

The authors emphasize that:

Geometric Grounding is Key: Strong geometric priors learned from tracking are more effective for metric motion forecasting than generative priors focused on pixel appearance.
Passive Learning is Sufficient: Complex interaction dynamics can be learned from passive, unlabelled human videos without the need for expensive 3D sensors or language annotations.
Generalization: The approach generalizes well to out-of-distribution objects, environments, and viewpoints, suggesting that the learned motion priors capture fundamental physical interactions rather than dataset-specific artifacts.

The work concludes that it is possible to transform retrospective video understanding into forward-looking geometric reasoning, providing a compact interface for downstream planning and embodied reasoning systems.