💻 computer science

Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation

The paper presents MoGe4D, a geometry-conditioned framework that synthesizes interactive 4D scenes from a single static image by predicting dense, time-varying point trajectories via a diffusion process, supported by a new large-scale dataset (TrajScene-60K) and specialized modules for motion normalization and view synthesis to ensure spatiotemporal coherence and structural stability.

Original authors: Yanran Zhang, Ziyi Wang, Wenzhao Zheng, Zheng Zhu, Jie Zhou, Jiwen Lu

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanran Zhang, Ziyi Wang, Wenzhao Zheng, Zheng Zhu, Jie Zhou, Jiwen Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a single, frozen photograph of a scene—maybe a bear walking on rocks or a surfer riding a wave. Your goal is to turn that still image into a full 3D movie that you can watch from any angle, with the objects moving naturally.

This is the challenge the paper MoGe4D tackles. The authors argue that most current methods try to do this in two separate, clumsy steps, which often leads to weird glitches (like objects melting or moving in impossible ways). Instead, MoGe4D tries to do everything in one smooth, connected process.

Here is a breakdown of how it works, using simple analogies:

1. The Problem: The "Bad Translator" vs. The "Architect"

The paper says existing methods usually follow one of two flawed paths:

Path A (Generate then Reconstruct): First, they try to make a 2D video of the scene moving, and then they try to "sculpt" a 3D shape out of that video.
- The Analogy: Imagine trying to build a house by first drawing a picture of the house on a piece of paper, and then trying to build the actual house based only on that drawing. The drawing might look great, but the 3D house ends up wobbly because the drawing didn't have strict rules about how the walls fit together.
Path B (Reconstruct then Generate): First, they build a static 3D model (like a statue), and then they try to animate it.
- The Analogy: This is like building a perfect clay statue of a dancer, but then trying to make the clay dance. The clay is stiff; it can't move naturally because the "dancing" part was added after the "building" part was finished.

MoGe4D's Solution: Instead of separating the "building" (geometry) from the "dancing" (motion), it treats them as a single, inseparable unit. It imagines the scene not as a solid object, but as a cloud of millions of tiny, invisible dots (points) that have a specific path they want to follow over time.

2. The Secret Sauce: The "Dense Trajectory"

Instead of guessing how a whole object moves, MoGe4D tracks the movement of every single pixel as a 3D dot.

The Analogy: Think of a swarm of fireflies. If you watch a firefly swarm, you don't just see "a blob moving." You see thousands of individual fireflies, each with its own path. MoGe4D predicts the path for every single "firefly" (point) in the image simultaneously. Because it knows where every dot is supposed to go, the whole shape stays solid and doesn't melt or warp.

3. The New Dataset: "TrajScene-60K"

To teach an AI to do this, you need a massive library of examples showing how 3D points move in real life. The authors realized no such library existed, so they built one called TrajScene-60K.

The Analogy: Imagine trying to teach a student to be a master chef, but you only have a textbook with pictures of food, not actual recipes or ingredients. The authors went out, collected 60,000 high-quality video clips, and for every single frame, they calculated exactly where every point in the scene was moving. They created a "cookbook" of 3D motion for the AI to study.

4. The Two Main Tools

The system uses two specialized tools to turn a photo into a 4D movie:

A. The "Motion Predictor" (4D-STraG)
This is the brain of the operation. It takes your photo and asks, "If this scene were a movie, how would every single dot move?"

Depth-Guided Normalization: The AI knows that a small movement by a close object looks huge on the screen, while the same movement by a far object looks tiny. This tool acts like a ruler, adjusting the scale of the movement so the AI learns the real physics, not just how things look on a flat screen.
Motion Perception Module (MPM): This is like a spotlight. It looks at the photo and says, "The bear's legs are likely to move, but the mountain in the background probably won't." It tells the AI where to focus its energy on creating movement, ensuring the background stays stable while the subject dances.

B. The "Camera Operator" (4D-ViSM)
Once the AI has predicted how all the dots move, this tool acts as the camera operator. It takes that cloud of moving dots and renders a video from any angle you want.

The Analogy: If the first tool built the 3D movie, this tool is the one holding the camera and walking around the set, filming the action from the left, the right, or even from above, filling in any gaps so the video looks smooth.

5. The Results

The paper claims that by keeping the "building" and "moving" steps tightly linked, their method produces:

Stable Shapes: Objects don't melt or twist into weird shapes.
Realistic Motion: Things move in a way that makes physical sense.
New Angles: You can watch the scene from angles that weren't in the original photo.

In short, MoGe4D is like a master puppeteer who doesn't just move the puppet's strings (motion) or carve the puppet's body (geometry) separately. Instead, it carves the puppet while simultaneously figuring out exactly how every joint will move, resulting in a performance that is both structurally sound and dynamically alive.

Technical Summary: MoGe4D – Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation

Problem Definition

Generating interactive, dynamic 4D scenes from a single static image is a fundamental challenge in computer vision. The core difficulty lies in inferring plausible future motion and 3D structure from a static observation while maintaining strict spatiotemporal coherence across time and viewpoints. Existing approaches typically decouple geometry and motion into two disconnected stages, leading to significant limitations:

Generate-then-Reconstruct: These methods use video generation models to synthesize multi-view videos before reconstructing 4D representations. While they offer high-fidelity dynamics, they often fail to maintain strict geometric consistency across views, resulting in artifacts and structural collapse during reconstruction.
Reconstruct-then-Generate: These methods first establish a static 3D structure and then animate it. While geometrically robust, they often discard the rich dynamic potential latent in the source image, restricting motion to simple, externally constrained animations (e.g., swinging) and struggling with self-initiated, large-scale movements.

Methodology

MoGe4D (Motion and Geometry-Aware image-to-4D Synthesis) proposes a unified framework that tightly couples geometric modeling and motion generation. Instead of treating them as separate stages, the method models the scene as dense 4D point trajectories predicted directly from an initial geometric prior inferred from the input image.

1. Dataset: TrajScene-60K

To address the scarcity of high-quality training data for scene-level 4D generation, the authors constructed TrajScene-60K, a dataset of 60,000 video samples.

Source: Curated from WebVid-10M using a two-stage automated filtering pipeline.
Filtering: Large Language Models (CogVLM2, DeepSeek-V3) filter for videos with countable entities and self-initiated, non-rigid, or articulated motion, discarding unstructured dynamics (e.g., crowds, wind) and camera-induced motion.
Annotation: Dense 4D point tracks are extracted using the DELTA model, refined via depth filtering, and rendered using Gaussian Splatting. The dataset includes over 3 million frames and 12 billion 3D point annotations with per-frame depth and language descriptions.

2. 4D Scene Trajectory Generator (4D-STraG)

This is the core diffusion-based module that predicts future time-varying point trajectories relative to an initial frame.

Architecture: Built upon the Wan2.1 diffusion transformer (DiT). It employs a lightweight Trajectory Encoder to convert relative point displacements into RGB motion maps and a Trajectory Decoder to recover trajectories.
Depth-Guided Motion Normalization: To handle scale ambiguity (where small 3D movements in near objects cause large 2D displacements), the method normalizes motion vectors relative to the viewing frustum size at the initial depth ( $z$ ). This transforms raw motion into a scale-invariant representation, allowing the model to learn motion patterns independent of absolute spatial position.
Motion Perception Module (MPM): To leverage motion priors from image-to-video models without introducing geometric inconsistencies, MPM identifies semantic regions likely to move. It uses a pretrained motion feature extractor (OmniMAE) and injects these features into the diffusion process via Motion-aware Adaptive Normalization (MAdaNorm). This enables token-wise modulation, ensuring fine-grained control over temporal coherence in a geometrically consistent manner.
Training Objective: The model uses flow matching to learn deterministic flows from noise to data distributions, minimizing the error between predicted and true flow fields.

3. 4D View Synthesis Module (4D-ViSM)

Once the dense 4D point cloud trajectories are generated, 4D-ViSM renders them into high-fidelity dynamic videos from arbitrary camera trajectories.

Rendering Strategy: It rasterizes each frame of the time-varying point cloud as 3D Gaussians. To handle holes caused by occlusion or sparse coverage in novel views, it leverages a finetuned video generation model (Wan2.1) with mask processing (setting mask values to 0.5 for unprojected regions) to complete missing regions, ensuring visual coherence.

4. Inference Pipeline

Given a single image and text prompt:

Estimate initial depth (UniDepthv2).
Encode image and depth into latents; extract motion features via MPM.
The DiT generates relative motion latents.
De-normalize motion using the depth-guided strategy and fuse with the initial point cloud.
Render the resulting 4D scene into novel-view videos using 4D-ViSM.

Key Contributions

TrajScene-60K: A large-scale dataset of 60,000 high-quality 4D scenes with dense point trajectories, addressing the data scarcity for complex scene-level 4D generation.
Geometry-Conditioned Formulation: A novel approach that predicts scene trajectories from an initial geometric prior, avoiding the reliance on loosely decoupled "generate-then-reconstruct" or "reconstruct-then-generate" stages.
4D-STraG & 4D-ViSM: A diffusion-based trajectory generator featuring depth-guided motion normalization and a Motion Perception Module for structural stability, coupled with a view synthesis module for high-quality novel-view rendering.
Empirical Validation: Extensive experiments demonstrating superior performance in temporal coherence, geometry-aware consistency, and visual fidelity compared to existing baselines.

Experimental Results

The authors evaluated MoGe4D against state-of-the-art methods (4Real, GenXD, DimensionX, Gen3C, Free4D) using both standard video benchmarks and custom 4D consistency metrics.

Video Quality (VBench): MoGe4D consistently outperformed baselines in Dynamic Degree (achieving 1.0000 in some groups), Aesthetic Quality, and Imaging Quality. It showed particularly strong performance in Subject Consistency and Background Consistency compared to optimization-based methods.
4D Consistency (VLM-based): Using Qwen2.5-VL-72B for evaluation, MoGe4D achieved the highest scores in 3D Geometric Consistency, Temporal Texture Stability, and Motion-Geometry Coupling, validating its ability to preserve structure during complex camera motions.
Geometric Accuracy: MoGe4D significantly reduced Average Trajectory Error (0.042–0.058) compared to baselines like Free4D (0.252) and Gen3C (0.197), while maintaining competitive 3D Reprojection Errors.
Efficiency: The method synthesizes 49-frame, 512×368 sequences in approximately 6 minutes on a single A100 GPU, which is 5× faster than Free4D under equivalent resolutions and significantly faster than optimization-based approaches that require hours.

Significance and Limitations

Significance:
The paper claims that MoGe4D establishes a new technical pathway for 4D synthesis from minimal input. By tightly coupling geometry and motion through dense trajectory generation, it overcomes the spatiotemporal inconsistencies inherent in decoupled paradigms. The work highlights the importance of large-scale, high-quality datasets (TrajScene-60K) and geometry-conditioned priors for achieving physically plausible and structurally stable 4D generation.

Limitations:
The authors acknowledge that the method struggles with extreme photometric conditions (e.g., harsh backlighting) and highly unstructured dynamics (e.g., chaotic crowds). In these edge cases, the reliance on off-the-shelf monocular depth estimators for the initial geometric prior can introduce noise, leading to less reliable motion generation. Future work is suggested to integrate more robust visual foundation models to handle such physical extremes.