MoRe: Motion-aware Feed-forward 4D Reconstruction Transformer

MoRe is a feed-forward 4D reconstruction transformer that efficiently recovers dynamic 3D scenes from monocular videos by employing an attention-forcing strategy to disentangle motion from static structure and utilizing grouped causal attention for temporally coherent geometry.

The Gist

Imagine you are trying to build a 3D model of a busy city street using only a video from your phone. The challenge? The street is full of moving cars, walking people, and fluttering flags.

Most computer vision systems get confused by this. They try to figure out where the camera is by looking at everything in the frame. When a car zooms by, the system thinks, "Oh, the whole world just moved!" and gets the 3D map all wrong. It's like trying to navigate a room while someone keeps shoving furniture around you; you lose your sense of direction.

MoRe (Motion-aware Feed-forward 4D Reconstruction Transformer) is a new AI system designed to solve this exact problem. Here is how it works, explained through simple analogies:

1. The "Smart Filter" (Motion-Forcing Attention)

Imagine you are a photographer trying to take a picture of a static statue in a park, but a dog is running wildly in front of it.

• Old AI: Tries to focus on the whole scene. The dog's movement blurs the statue, and the camera gets shaky.
• MoRe: Has a special "Smart Filter" trained to ignore the dog. During its training, teachers showed it exactly where the moving objects were (using motion masks) and said, "Don't look at the dog; look at the statue and the trees."
• The Result: When MoRe watches a video, it learns to mentally "blur out" the moving cars and people, focusing only on the static buildings and roads to figure out where the camera is. This keeps the 3D map stable even when chaos is happening around it.

2. The "One-Way Street" (Grouped Causal Attention)

Usually, AI models look at a video like a book: they read the whole thing at once to understand the story. But if you are watching a live stream, you can't read the last page before the first one happens!

• The Problem: Traditional models get overwhelmed if the video is too long, like trying to remember every word of a 2-hour movie instantly.
• MoRe's Solution: It treats the video like a one-way street. It processes the video frame-by-frame, remembering what it saw in the past but never peeking at the future.
• The "Grouped" Twist: Inside a single frame (a single photo), all the pixels can talk to each other freely (like a group chat) to understand the shape of a building. But between frames (from one second to the next), they only talk in one direction (like a relay race). This makes it incredibly fast and perfect for streaming video.

3. The "Group Hug" (Bundle Adjustment Refinement)

Even with a one-way street, if you walk for a long time, you might start to drift slightly off course.

• The Issue: As MoRe processes a long video, tiny errors can add up, making the 3D map look a bit warped at the end.
• MoRe's Solution: Once the video is processed, MoRe does a quick "Group Hug." It looks back at all the data it just collected and gently adjusts the camera's path and the map's shape to make everything fit together perfectly. It's like a GPS recalculating your route after a long drive to ensure you ended up exactly where you thought you were.

Why is this a big deal?

• Speed: It's a "feed-forward" system, meaning it doesn't need to run slow, repetitive calculations. It sees the video and instantly outputs the 3D map, like a human recognizing a face instantly rather than analyzing every feature one by one.
• Versatility: It works on static scenes (like a quiet room) just as well as dynamic ones (like a busy highway).
• Real-Time Ready: Because it's so efficient, it could eventually power things like:
  • Augmented Reality (AR): Putting virtual furniture in your living room while your cat runs around.
  • Robotics: Helping robots navigate a factory floor with moving forklifts.
  • Digital Twins: Creating accurate 3D copies of real-world cities for planning, even with traffic moving.

In short: MoRe is a super-smart, fast 3D scanner that knows how to ignore the chaos of moving objects to build a perfect, stable map of the world, all while watching a live video stream.

Technical Summary

1. Problem Statement

Reconstructing dynamic 4D scenes (time-evolving 3D structures) from monocular videos is a significant challenge in computer vision. Existing approaches face a critical trade-off:

• Optimization-based methods (e.g., SLAM, bundle adjustment) handle dynamic scenes well but are computationally expensive, iterative, and struggle with real-time streaming or long sequences.
• Feed-forward deep learning methods are fast but are typically trained on static scenes. They fail in dynamic environments because moving objects corrupt camera pose estimation and depth prediction, leading to geometric inconsistencies.
• Current Transformer-based solutions often suffer from quadratic inference costs regarding sequence length (making streaming impractical) and lack explicit mechanisms to disentangle dynamic motion from static scene geometry.

The core problem addressed by MoRe is: How to design a fast, generalizable, feed-forward framework that handles both camera and object motion in streaming video inputs while producing accurate camera poses, depths, and 3D geometry without requiring explicit motion priors during inference.

2. Methodology

MoRe is a motion-aware, feed-forward Transformer network designed for streaming 4D reconstruction. It integrates three core innovations:

A. Motion-Aligned Attention (Training Strategy)

The central innovation is an attention-forcing strategy that teaches the model to distinguish between static background and dynamic objects purely through training supervision.

• Mechanism: During training, the model utilizes ground-truth motion masks (derived from datasets or extracted via SAM2/flow discrepancy). These masks are converted into "motion scores" for image tokens.
• Supervision: The attention weights of the "camera token" (which estimates pose) are explicitly supervised to align with these motion scores. The model is penalized if it attends to moving objects when estimating camera pose.
• Inference Benefit: This allows the model to implicitly learn to ignore dynamic regions during inference. No motion masks or segmentation priors are required at test time, keeping the inference lightweight and end-to-end.

B. Grouped Causal Attention (Streaming Architecture)

To support real-time streaming inputs, MoRe replaces standard global attention with a Grouped Causal Attention mechanism.

• Frame-wise Causality: Unlike standard causal attention (which treats all tokens as a flat sequence), MoRe enforces causality between frames but allows full bidirectional attention within a single frame. This preserves spatial coherence and geometric reasoning within each image while maintaining temporal causality.
• KV Caching: The model caches Key-Value (KV) pairs from previous frames. New frames attend to the accumulated history, enabling efficient online processing with constant latency.

C. BA-like Token Aggregation (Refinement)

To address the limitation of strictly causal attention (which restricts long-term global information exchange), MoRe introduces a post-processing refinement step analogous to Bundle Adjustment (BA).

• Process: After the streaming inference is complete, the camera tokens are duplicated and perform an additional attention pass over all cached features from the entire sequence.
• Goal: This lightweight global refinement step corrects accumulated drift and ensures temporal consistency across long sequences without sacrificing the efficiency of the initial streaming pass.

D. Training Objective

The model is trained end-to-end on a massive, diverse dataset (static and dynamic) using a multi-task loss:

• Confidence-weighted regression for depth and point maps.
• Binary Cross-Entropy (BCE) for motion mask prediction.
• Attention Alignment Loss: A custom loss to force camera token attention to align with static regions.
• Camera Pose Loss: Relative transformation loss applied to both the streaming path and the post-hoc aggregated path to ensure consistency.

3. Key Contributions

1. MoRe Framework: A unified, feed-forward 4D reconstruction system capable of jointly estimating camera poses, depths, point maps, and motion masks from monocular video.
2. Attention-Forcing Strategy: A novel training technique that explicitly disentangles dynamic motion from static structure, enabling robust reconstruction in dynamic scenes without explicit motion priors at inference.
3. Streaming Inference Mechanism: A design combining Grouped Causal Attention (for spatial-temporal consistency) and BA-like Token Aggregation (for global refinement), enabling efficient, real-time processing of long video sequences.
4. State-of-the-Art Performance: Extensive experiments demonstrate that MoRe achieves superior accuracy in both dynamic and static scenarios compared to existing optimization-based and feed-forward methods.

4. Experimental Results

MoRe was evaluated on multiple benchmarks, including Sintel, TUM-dynamics, Bonn, ScanNet, and Co3Dv2.

• Camera Pose Estimation: MoRe significantly outperforms streaming baselines (e.g., Stream3R, CUT3R, StreamVGGT) in Absolute Translation Error (ATE) and Relative Rotation Error (RPE). For instance, on the dynamic Sintel dataset, MoRe achieved an ATE of 0.1474, compared to 0.2163 for CUT3R and 0.4159 for StreamVGGT. It also rivals full-attention models (like VGGT) while being much faster.
• Depth Estimation: The model achieves state-of-the-art results in video depth estimation, showing strong generalization across synthetic and real-world datasets (e.g., 0.254 Abs-Rel on Sintel for the streaming version).
• Efficiency: MoRe achieves an inference speed of 30.09 FPS on KITTI (512x144 resolution), outperforming most baselines while maintaining high reconstruction quality.
• Generalization: The model demonstrates zero-shot generalization on unseen dynamic datasets and maintains high performance on static scenes (Co3Dv2), proving that the motion-aware design does not compromise static reconstruction capabilities.

5. Significance

MoRe represents a significant leap forward in real-time 4D reconstruction.

• Practicality: By eliminating the need for iterative optimization and explicit motion segmentation during inference, it makes high-quality dynamic 3D reconstruction feasible for real-world applications like Augmented Reality (AR), Robotics, and Digital Twins.
• Robustness: It solves the "dynamic object corruption" problem that plagues previous feed-forward models, allowing systems to operate reliably in busy, moving environments.
• Scalability: The streaming architecture allows for the processing of arbitrarily long video sequences with constant memory and latency growth, a critical requirement for continuous monitoring and autonomous navigation.

In summary, MoRe bridges the gap between the speed of feed-forward models and the robustness of optimization-based methods, providing a scalable solution for reconstructing the evolving 3D world.