💻 computer science

Forge4D: Feed-Forward 4D Human Reconstruction and Interpolation from Uncalibrated Sparse-view Videos

Forge4D is a feed-forward model that enables efficient, instant reconstruction and interpolation of dynamic 3D humans from uncalibrated sparse-view videos by jointly optimizing streaming 3D Gaussian reconstruction with self-supervised dense motion prediction and occlusion-aware fusion.

Original authors: Yingdong Hu, Yisheng He, Jinnan Chen, Weihao Yuan, Kejie Qiu, Zehong Lin, Siyu Zhu, Zilong Dong, Steven Hoi, Jun Zhang

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yingdong Hu, Yisheng He, Jinnan Chen, Weihao Yuan, Kejie Qiu, Zehong Lin, Siyu Zhu, Zilong Dong, Steven Hoi, Jun Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to capture a person moving through a room and then instantly recreating that movement from any angle, even one no camera ever saw, or at a moment in time that was never recorded. This is the dream of four-dimensional reconstruction: building a living, breathing digital twin that exists not just in space, but in time. For years, achieving this has required either a studio full of synchronized cameras or hours of slow, computer-heavy processing that makes real-time interaction impossible. The goal has always been to turn a few scattered video clips into a complete, fluid 3D world where a person can be viewed from anywhere, at any moment, without the digital model falling apart or looking like a blurry mess.

A team of researchers has now taken a significant step toward making this dream a reality with a new system called Forge4D. Instead of trying to solve the entire problem of moving 3D shapes all at once, which often leads to confusion and errors, they broke the task down into two simpler, connected jobs. First, the system builds a static 3D model of the person and their surroundings from a few uncalibrated video streams. Think of this as creating a high-quality, frozen sculpture of the scene. Second, the system predicts exactly how every tiny part of that sculpture moves from one second to the next. By separating the shape from the motion, the researchers found a way to make the process fast enough to be useful in real time, even when the cameras recording the scene are not perfectly aligned or calibrated.

The core of their discovery lies in how they handle the data. Previous methods often tried to process an entire video sequence at once, which overwhelmed computer memory and slowed everything down. Forge4D takes a different approach by treating the video as a stream. It processes the frames one after another, using a special memory tool to remember what happened in the previous moments without needing to reload the entire history. This allows the system to maintain a consistent sense of scale and position as the person moves, ensuring that the digital model doesn't shrink, grow, or drift away as the video plays. It is a method that keeps the reconstruction stable and efficient, even when the input data is sparse and imperfect.

Once the system has a stable 3D model, it faces the challenge of predicting motion. Since there is no perfect map of how every point on a person's body moves in the real world to use as a teacher, the researchers invented a new way to teach the computer. They asked the system to guess the motion, then tried to warp the 3D model forward in time based on that guess. If the warped model looked like the actual next frame of the video, the guess was good. If it looked wrong, the system adjusted its understanding of the motion. This self-correcting loop, combined with a check against how light and shadow usually behave, allowed the computer to learn the complex dance of human movement without needing a pre-written script. It also developed a way to handle parts of the body that are hidden from view, ensuring that the digital model remains solid and does not flicker or glitch when a person turns or an object blocks the view.

The results of this approach are striking. When tested on various datasets, including complex scenes with people interacting with objects, the system produced images that were sharper and more realistic than those generated by previous methods. It could create new views of a scene from angles that were never filmed, and it could generate smooth, natural-looking movement for moments in time that were never captured. In tests, the system was able to produce these high-quality images in less than a quarter of a second per frame, a speed that opens the door to interactive applications like virtual reality, live broadcasting, and immersive communication. The researchers noted that while the system works exceptionally well for normal movements, it can struggle if the motion is extremely fast or if the time gap between frames is too large, suggesting that the assumption of smooth, continuous movement has its limits.

Ultimately, this work demonstrates that it is possible to reconstruct dynamic human scenes from simple, uncalibrated videos with a speed and quality that was previously out of reach. By simplifying the problem into manageable steps and teaching the computer to learn from its own predictions, the researchers have created a tool that brings us closer to a future where digital avatars can be generated instantly and interacted with in real time. The system does not just capture a moment; it understands the flow of time within that moment, allowing us to step into the video and look around as if we were truly there.

Technical Summary: Forge4D

Problem Statement

The paper addresses the challenge of instant 4D human-object reconstruction and interpolation from uncalibrated sparse-view videos. This capability is critical for downstream applications such as real-time livestreaming, sports broadcasting, AR/VR, and immersive holographic communication.

Existing methods face two primary limitations:

Iterative Optimization Approaches: Methods that rely on optimizing 4D representations over dense-view sequences suffer from slow reconstruction speeds, require calibrated camera parameters, and cannot generate novel-time representations efficiently.
Feed-Forward Limitations: While recent feed-forward models (e.g., VGGT, NoPoSplat) can reconstruct static 3D scenes or point clouds from uncalibrated inputs, they are generally incapable of handling dynamic scenes, synthesizing novel-time images, or maintaining metric-scale consistency across time.

The core difficulty lies in simultaneously recovering accurate human-object geometry and dense motion trajectories from unposed, sparse inputs while maintaining real-time interactivity.

Methodology: Forge4D

The authors propose Forge4D, the first feed-forward model capable of metric-scale 4D human-object reconstruction and interpolation from uncalibrated sparse-view videos. The framework decomposes the complex 4D problem into two sequential tasks: streaming 3D Gaussian reconstruction and dense motion prediction. The training pipeline consists of three stages:

Stage 1: Feed-Forward Static 3D Gaussian Reconstruction

The model leverages a pre-trained backbone (VGGT) to recover geometry and camera poses but adapts it to predict 3D Gaussian Splatting (3DGS) assets instead of point clouds for photorealistic rendering.

Metric Gauge Alignment: A fundamental challenge is the scale discrepancy between the normalized output of VGGT and real-world metric scales. The authors introduce a metric gauge regularization ( $p_{gauge}$ ) that aligns the model's internal coordinate system with ground-truth camera extrinsics. This is achieved by calculating the ratio of translation magnitudes between predicted and ground-truth cameras, ensuring stable optimization and enabling metric measurements.
Loss Functions: The model is supervised using a combination of photometric losses (L2, SSIM, LPIPS) on input and novel views, and a comprehensive camera loss that enforces accurate rotation, translation direction, and consistent relative scaling.

Stage 2: Streaming Dynamic Reconstruction with State Tokens

To handle video streams without incurring Out-of-Memory (OOM) errors or excessive latency from global attention over all frames:

State Tokens: The authors introduce learnable state tokens that iteratively encode information from previous frames. These tokens act as Key and Value in a cross-attention mechanism for the current frame, while the current frame updates the token as a Query. This allows for efficient temporal information propagation in a memory-friendly manner.
Temporal Consistency: The metric gauge regularization is extended to a temporal form, ensuring consistent global scale and camera parameters across the entire video sequence.

Stage 3: Novel-Timestamp Synthesis via Dense Motion Prediction

To enable interpolation to arbitrary intermediate timestamps, the model predicts dense motion fields for each 3D Gaussian.

Dense Motion Prediction: A dedicated module predicts a dual motion field (backward and forward) for each 3D Gaussian between adjacent frames.
Self-Supervised Retargeting Loss: Since ground-truth dense 3D motion is unavailable, the authors formulate motion prediction as a 3D point-matching task. They deform the 3D Gaussians of the current frame to the previous frame using the predicted motion and supervise the rendered result against the ground-truth image of the previous frame.
Occlusion-Aware Optical Flow Loss: To further regularize the motion and ensure physical plausibility, an optical flow loss is introduced. It uses pseudo-ground-truth flows (from SEA-RAFT) and a cyclic consistency mask to penalize inconsistencies and handle occlusions.
Gaussian Fusion for Interpolation: For novel-time synthesis, the 3D Gaussians from two adjacent frames are warped to the intermediate timestamp under a constant velocity assumption. An occlusion-aware fusion MLP merges these deformed representations, filtering out occluded points and reducing temporal redundancy to prevent flickering.

Key Contributions

First Feed-Forward 4D Model: Forge4D is the first model to perform metric-scale 4D human-object reconstruction and novel-time interpolation from uncalibrated sparse-view videos in a single forward pass.
Decomposed Architecture: The work simplifies 4D reconstruction by decoupling it into streaming 3DGS prediction and dense motion estimation, stabilized by novel metric gauge, retargeting, and occlusion-aware flow losses.
Motion-Guided Fusion: A novel Gaussian fusion method is introduced for 3D Gaussian interpolation, effectively mitigating flickering and jittering artifacts caused by temporal redundancy.

Experimental Results

The model was evaluated on the DNA-Rendering (in-domain), Genebody (out-of-domain), and SelfCap/VVC (in-the-wild) datasets.

Novel-View Synthesis: Forge4D outperforms both optimization-based methods (e.g., DualGS, D-3DGS) and existing feed-forward models (e.g., NoPoSplat, AnySplat, GPS-Gaussian). On the DNA-Rendering dataset, it achieves a PSNR of 29.82 dB, surpassing the previous best feed-forward method (GPS-Gaussian) by +2.90 dB.
Novel-Time Synthesis: In 4D interpolation tasks, Forge4D significantly outperforms the state-of-the-art feed-forward 4D model L4GM, achieving a PSNR gain of up to +12.57 dB on the Genebody dataset.
Metric Scale: The model achieves a mean point distance error of 0.0264m against ground-truth meshes, outperforming MoGe-2 (0.3309m).
Efficiency: The full model processes a frame pair in 224.27 ms. With interpolation, it can generate 10 intermediate frames in an additional 1.46 ms, resulting in an effective output frame rate of 44.42 FPS.

Significance and Limitations

The paper claims that Forge4D represents a significant step forward by enabling real-time, interactive 4D reconstruction without the need for camera calibration or iterative optimization. It bridges the gap between static feed-forward reconstruction and dynamic scene understanding, offering high-fidelity novel-view and novel-time synthesis.

The authors acknowledge limitations: performance degrades with large motions or long inter-frame intervals due to reduced inter-frame correspondences and violations of the constant velocity assumption. Future work is suggested to focus on improving motion modeling under extreme displacements and optimizing computational efficiency.