💻 computer science

OmniX: Any-view and Any-time 4D Reconstruction via Feed-forward Trajectory Fields

OmniX is a feed-forward 4D reconstruction framework that predicts dense 3D point trajectories for videos with large camera motions by decoupling dynamic motion modeling from static geometry using compact dynamic tokens, supported by a newly introduced large-scale synthetic dataset for training.

Original authors: Yanqin Jiang, Tengfei Wang, Zhengwei Wang, Chenjie Cao, Junta Wu, Wenhan Luo, Weiming Hu, Jin Gao, Chunchao Guo

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanqin Jiang, Tengfei Wang, Zhengwei Wang, Chenjie Cao, Junta Wu, Wenhan Luo, Weiming Hu, Jin Gao, Chunchao Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to recreate a bustling city square in a video game, but you only have a shaky handheld camera recording people walking, cars zooming by, and the sun moving across the sky. The camera swings wildly, zooms in, and jumps around. Most previous attempts to build a 3D model from this kind of video were like trying to solve a puzzle while wearing blinders: they either froze the moving people (ignoring the action) or got confused when the camera moved too much, resulting in a blurry, broken mess.

Enter OmniX, a new "feed-forward" framework (think of it as a super-fast, one-shot recipe) that finally manages to reconstruct these dynamic scenes in 4D—meaning it captures the 3D shape plus how every single point moves over time, even when the camera is going crazy.

The Magic Trick: Separating the Static from the Spinning

The paper argues that previous methods failed because they tried to figure out the building's shape and the person's dance moves at the same time, without separating them. It's like trying to listen to a solo violin while a drum solo is happening, without a mixer to separate the tracks.

OmniX fixes this by explicitly disentangling (unmixing) the two. It treats the background (static geometry) and the foreground (dynamic motion) as different ingredients.

The Static Part: It figures out where the walls and ground are.
The Dynamic Part: Instead of tracking every single pixel's movement individually (which is slow and computationally heavy), OmniX uses a clever trick. It identifies a tiny, sparse set of "dynamic tokens." Think of these as a few super-scouts sent out to map the movement of the whole crowd. Because 3D motion is naturally sparse and organized (low-rank), these few scouts can generate a "trajectory field"—a map of how every pixel in the video should move, from every angle and at every moment in time.

The "One-Pass" Superpower

Older methods often worked like a slow-motion detective, checking frame by frame, asking "Where did this point go?" and then "Where did it go next?" over and over again. This is called "iterative," and it's slow.

OmniX is different. It predicts the entire path of every point in one single pass. It's like watching a movie and instantly knowing the entire future path of every character without having to pause and rewind. This is made possible by a new mechanism called Sparse Spatiotemporal Attention (SSA). The SSA picks the most important "dynamic tokens," expands them across time, and lets them "chat" with all the image data to figure out the motion. Then, a Deformable Trajectory Sampling Head takes those sparse clues and fills in the gaps to give you a dense, pixel-perfect trajectory for the whole scene.

Building the Training Gym: The UE5 Engine

You can't teach a robot to dance if it's never seen a dance floor. The authors realized there wasn't enough real-world data with "large camera motion" and perfect 3D labels to train a model this ambitious. So, they built their own gym.

Using Unreal Engine 5 (UE5), they created an automatic data engine. They took static environments, threw in dynamic objects (like moving cars and people), and filmed them with a custom camera system that swung around wildly.

The Result: They generated a massive dataset of 80,000 scenes and 1.28 million multi-view videos.
The Annotation: Every single video comes with perfect "ground truth" labels: exact depth, camera poses, and dense 3D point trajectories. This synthetic data is what allowed OmniX to learn how to handle those wild, 180-degree camera swings that confused previous models.

The Results: How Good Is It?

The paper tested OmniX on several challenges, and the numbers suggest it's a new leader in the field.

Dense 3D Point Trajectory: On their custom validation set, OmniX beat the previous best methods (like VDPM and TraceAnything) by a significant margin. For example, on "Disjoint Video Pairs" (where you have two separate video clips that don't overlap in time), OmniX achieved an APD3D (Average Point Distance) of 0.444 for all points, compared to 0.306 for the runner-up VDPM. (Note: In this specific metric, a higher APD3D indicates better performance). The error rates also dropped significantly, with OmniX achieving an EPE (End-Point Error) of 0.101 compared to 0.306 for VDPM in some settings.
3D Point Tracking: On the TAPVid-3D benchmark, OmniX achieved state-of-the-art results across three different datasets (ADT, DriveTrack, PStudio). For instance, on the ADT dataset, it reached an APD3D of 0.367, beating the next best method (VDPM at 0.266) by a large margin. Since higher APD3D is better here, this represents a substantial improvement in tracking accuracy.
Other Tasks: It also did very well on video depth estimation and camera pose estimation, matching or beating top competitors on datasets like KITTI and Sintel.

What It's NOT (And What It Rules Out)

The paper is very clear about what doesn't work, based on their experiments:

No "Self-Attention" for the Scouts: They tried adding a "self-attention" layer (where the dynamic tokens talk to each other) in their Sparse Spatiotemporal Attention module. The result? It made almost no difference. The paper explicitly rules this out as unnecessary, saving computing power by removing it.
No Iterative Guessing: The paper argues against the old way of "iteratively querying" points. Their "one-pass" approach is proven to be faster and more accurate for dense predictions.
No "Small Camera Motion" Limit: Previous methods that predicted dense trajectories were limited to videos where the camera barely moved. OmniX explicitly proves it works with large camera motions (up to 180 degrees) and even temporally disjoint inputs (videos that don't overlap in time).

The Bottom Line

OmniX suggests that by separating the "what is moving" from "what is standing still," and by training on a massive, synthetic dataset of 80,000 scenes, we can finally reconstruct dynamic 4D worlds from chaotic, handheld videos in a single, efficient pass. It's not just a small improvement; the authors show it sets a new state-of-the-art for predicting how every point in a scene moves, making it a powerful tool for things like autonomous driving and AR/VR content creation.

The paper doesn't claim this is a solved problem for every possible scenario in the universe, but the evidence from their 80k scenes and 1.28M videos strongly suggests this is the most robust method we have right now for handling large, complex camera movements.

Technical Summary: OmniX - Any-view and Any-time 4D Reconstruction via Feed-forward Trajectory Fields

1. Problem Statement

Current feed-forward 4D reconstruction methods face two primary limitations:

Motion Modeling: Many approaches predict per-frame static point clouds without explicitly modeling motion across time, requiring computationally expensive iterative tracking to recover temporal correspondences.
Viewpoint and Motion Constraints: Methods that do predict dense 3D point trajectories are typically restricted to monocular videos with small camera motions. They struggle under large viewpoint changes because they fail to disentangle static geometry feature matching from dynamic foreground temporal correspondence learning, leading to mutual interference.

There is a lack of frameworks capable of predicting dense 3D point trajectories from videos with large camera motions and temporally discontinuous inputs in a single feed-forward pass.

2. Methodology: OmniX Framework

OmniX is a feed-forward 4D reconstruction framework designed to recover consistent 4D geometry from arbitrary viewpoints and times. It explicitly disentangles dynamic foreground motion from static scene structure.

2.1 Formulation

The model takes a collection of videos as input and outputs:

3D Geometry: Depth maps, ray maps, and camera parameters (extrinsics and intrinsics) to recover 3D points.
4D Geometry: Dense 3D point trajectories over time.
- Trajectories are modeled as a transformation $\tau$ (rotation $A$ and translation $b$ ) applied to a reference 3D point $P^{(i_0)}_{j_0}$ to obtain its position at any time $(i, j)$ : $P^{(i)}_j = A^{(i)}_j P^{(i_0)}_{j_0} + b^{(i)}_j$ .
- To ensure efficiency, motion is represented via a trajectory field, where the per-pixel transformation is a weighted combination of a small set of learned trajectory transformation bases.

2.2 Architecture

OmniX builds upon feed-forward 3D reconstruction models (specifically DepthAnything3) and introduces a Trajectory Module consisting of two key components:

Sparse Spatiotemporal Attention (SSA):
- Dynamic Token Selection: An MLP predicts dynamic probabilities for image tokens, selecting the top- $\rho\%$ as "dynamic tokens" representing moving foregrounds.
- Temporal Expansion: These dynamic tokens are expanded along the temporal dimension using sinusoidal encodings of video and frame indices, modulated by per-token coefficients.
- Cross-Attention: The expanded dynamic tokens act as queries in a cross-attention mechanism, attending to all image tokens (keys/values) to extract geometric and motion cues. This allows the model to establish spatiotemporal correspondences and predict trajectory transformation bases efficiently. Self-attention among the query tokens is omitted as it provides negligible benefit.
Deformable Trajectory Sampling Head (DTSH):
- To convert the sparse trajectory field (predicted at token locations) into dense per-pixel transformations, the DTSH uses a deformable sampling mechanism based on the DPT architecture.
- It aggregates the sparse field via statistical (mean/variance) and salient (max pooling) branches, combined with dynamic scores.
- The head predicts sampling offsets and weights to upsample the sparse field to pixel-level transformation maps.
- Differentiable Selection: A dynamic score $\theta$ is predicted and used to modulate the final trajectory, encouraging background points to remain static ( $\hat{P}_{traj} = P + \theta \cdot (P_{traj} - P)$ ).

2.3 Training Objectives

The model is supervised using confidence-aware losses for depth, ray maps, and point trajectories to handle outliers (e.g., occlusions, sky).

Regression Loss: Weighted by predicted confidence to re-weight errors based on uncertainty.
Gradient Loss: Preserves sharp edges in depth maps.
Camera Pose Loss: Uses Huber loss for robustness.
Dynamic Score Loss: Binary Cross-Entropy (BCE) loss is applied to token-level and pixel-level dynamic scores when ground-truth dynamic masks are available.

3. Key Contributions

3.1 The OmniX Framework

The paper proposes a novel architecture that separates dynamic motion modeling from static geometry prediction. By leveraging the sparse and low-rank structure of 3D motion, it uses a small set of dynamic tokens to generate trajectory fields for all pixels across all views and times in a single pass. This enables robust performance under large camera motions and temporally discontinuous inputs.

3.2 Large-Scale 4D Dataset and Engine

Addressing the scarcity of 4D datasets with large camera motion and full geometric annotations, the authors developed an automatic data engine based on Unreal Engine 5 (UE5).

Synthesis: Composites static environments with dynamic objects and renders them using a customized multi-camera system.
Scale: The resulting dataset contains 80,000 scenes and 1.28 million multi-view videos.
Annotations: Includes full geometric annotations (depth, camera poses) and dense 3D point trajectories.

3.3 State-of-the-Art Performance

OmniX achieves state-of-the-art results on dense 3D point trajectory prediction and 3D point tracking. It also demonstrates competitive performance on video depth estimation and camera pose estimation tasks.

4. Experimental Results

4.1 Dense 3D Point Trajectory Prediction

Evaluated on a custom benchmark (600 videos from 40 scenes) with metrics End-Point Error (EPE) and Average Point Distance (APD3D).

Performance: OmniX significantly outperforms previous methods (e.g., TraceAnything, VDPM) across Monocular Video, Disjoint Video Pairs, and Hybrid Image-Video sets.
Large Motion: The model shows particular strength in scenarios with large camera motions (up to 180°) and disjoint video pairs, where baseline methods often fail due to training data limitations.
Foreground vs. All Points: The model achieves high accuracy on both foreground and all points, whereas baselines often degrade on "all points" due to inaccurate camera estimation causing background points to be projected out of view.

4.2 Sparse 3D Point Tracking

On the TAPVid-3D benchmark, OmniX achieves SOTA performance across ADT, DriveTrack, and PStudio datasets.

Qualitative Analysis: In complex scenes (e.g., walking, leg-crossing), OmniX correctly captures non-rigid motion patterns, whereas baselines often fail to distinguish foreground from background, resulting in warping artifacts or spurious background displacements.

4.3 Other Tasks

Video Depth Estimation: Achieves competitive results on KITTI, comparable to SOTA methods like DepthAnythingV3.
Camera Pose Estimation: Delivers strong performance on Sintel and TUM-dynamic datasets.

4.4 Efficiency

Inference: OmniX achieves the second-best efficiency among compared methods. While it uses more memory than simple tracking heads (due to native single-pass prediction of all trajectories), it avoids the iterative inference loops required by methods like VDPM and St4RTrack, resulting in faster runtimes than those iterative approaches.
Ablation: Removing the Cross-Attention mechanism causes a drastic performance drop, confirming its necessity for capturing foreground dynamics. Self-attention among dynamic tokens was found to be redundant.

5. Significance and Claims

The paper claims that OmniX represents a significant step forward in dynamic scene understanding by:

Solving the Large Motion Problem: Explicitly disentangling dynamic motion from static geometry allows for robust reconstruction under large camera motions and arbitrary viewpoints, a limitation of prior feed-forward methods.
Enabling Dense Prediction: The framework enables efficient, single-pass prediction of dense 3D point trajectories, overcoming the computational bottlenecks of iterative tracking.
Data Scalability: The introduction of the UE5-based data engine and the resulting 80k-scene dataset addresses the critical bottleneck of data scarcity for 4D reconstruction, demonstrating that larger, high-quality synthetic datasets directly translate to improved model accuracy.

The authors conclude that their framework and data generation pipeline provide a strong foundation for future research in embodied AI, autonomous driving, and AR/VR content creation.