💻 computer science

4DSloMo: 4D Reconstruction for High Speed Scene with Asynchronous Capture

This paper presents 4DSloMo, a high-speed 4D reconstruction system that achieves equivalent frame rates of 100–200 FPS using standard 25 FPS cameras through an asynchronous capture scheme and a novel video-diffusion-based generative model to correct artifacts from sparse viewpoints.

Original authors: Yutian Chen, Shi Guo, Tianshuo Yang, Lihe Ding, Xiuyuan Yu, Jinwei Gu, Tianfan Xue

Published 2026-06-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yutian Chen, Shi Guo, Tianshuo Yang, Lihe Ding, Xiuyuan Yu, Jinwei Gu, Tianfan Xue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to film a hummingbird's wings in slow motion. To see the wings clearly, you need a camera that takes hundreds of pictures every second. But most regular cameras only take 25 or 30 pictures a second. If you try to film the hummingbird with a slow camera, the wings just look like a blurry mess, and if you try to reconstruct the 3D shape of the bird from those blurry pictures, it falls apart.

This paper, titled 4DSloMo, solves this problem with a clever two-part trick: a hardware hack and a software magic spell.

Part 1: The Hardware Hack – "The Relay Race"

Usually, when scientists use a group of cameras to film a 3D scene, they tell all the cameras to snap a photo at the exact same time (synchronously). If you have 4 cameras, you get 4 photos per second.

The authors realized they could cheat the system without buying expensive, super-fast cameras. Instead of telling all cameras to snap at the same time, they told them to snap at slightly different times, like runners in a relay race passing a baton.

The Analogy: Imagine a line of 4 people standing in a row. If they all clap at the same time, you hear one big "clap." But if they clap one after another, very quickly, it sounds like a rapid-fire machine gun.
The Result: By staggering the start times of their cameras, they turned a standard 25-frames-per-second setup into a system that feels like it's capturing 100 to 200 frames per second. They didn't need new, expensive cameras; they just changed the timing.

Part 2: The Software Magic – "The Art Restorer"

There was a catch to this timing trick. Because the cameras were snapping at different times, at any single specific moment, the system only had a few cameras looking at the scene instead of all of them. This is like trying to paint a portrait but only having a few paintbrushes instead of a full set. The result was a 3D reconstruction full of "floaters" (ghostly, floating artifacts) and missing details.

To fix this, the authors trained a special AI artist (a video diffusion model).

The Analogy: Imagine you have a sketch of a moving dancer that is a bit shaky and has some smudges because you didn't have enough reference photos. You hand this sketch to a master painter who has seen thousands of videos of dancers. The painter doesn't just fix one frame; they look at the whole movie to understand how the dancer's arm moves from one second to the next. They fill in the missing details and smooth out the ghostly smudges, ensuring the dancer looks solid and moves naturally.
The Innovation: Previous AI tools tried to fix each frame individually, which made the dancer look like they were jittering or changing shape randomly between frames. This new AI looks at the video as a whole, keeping the motion smooth and consistent while cleaning up the mess.

The Final Result

By combining the "Relay Race" camera timing with the "Art Restorer" AI, the team could:

Capture fast, complex movements (like a person dancing or a piece of cloth flying) using cheap, standard cameras.
Reconstruct these scenes in high-quality 3D, even though the cameras were technically "slow."
Create a new dataset of real-world fast-motion videos to help others test their own ideas.

In short, they figured out how to make slow cameras act like super-fast ones, and then used a smart AI to clean up the blurry mess that the timing trick created, resulting in crystal-clear 3D movies of fast-moving scenes.

Technical Summary: 4DSloMo

Problem Statement

Reconstructing fast-dynamic scenes from multi-view videos is essential for applications in sports analysis, autonomous driving, and VR/AR content creation. However, current 4D capture systems are typically limited to frame rates below 30 FPS (e.g., 15–30 FPS), which is insufficient to capture high-speed motions like cloth movement or rapid athletic actions. Directly reconstructing high-speed motion from low-FPS inputs often leads to significant artifacts and motion distortions because existing methods cannot accurately interpolate intermediate frames for complex, non-linear motion.

Increasing the frame rate via specialized high-speed cameras is costly and increases data transmission bandwidth requirements. Conversely, purely algorithmic frame interpolation (e.g., using CNNs or standard diffusion models) often fails to recover the necessary physical information for rapid motion, resulting in severe artifacts or incorrect motion trajectories.

Methodology

The authors propose 4DSloMo, a framework that combines a novel hardware capture scheme with a software-based generative refinement pipeline to achieve high-speed 4D reconstruction using standard low-FPS cameras.

1. Asynchronous Capture Scheme

To increase the effective frame rate without new hardware, the authors introduce an asynchronous capture strategy.

Mechanism: Instead of synchronizing all cameras to capture simultaneously, the system staggers the start times of cameras within a group.
Implementation: Cameras are grouped (e.g., $K$ cameras per group). The $i$ -th camera in a group starts capturing at $t_i = i \cdot (\tau/K) + j \cdot \tau$ , where $\tau$ is the base frame interval (e.g., 1/25s).
Result: This effectively increases the temporal resolution by a factor of $K$ . For instance, using eight 25 FPS cameras divided into groups allows the system to achieve an equivalent capture rate of 100–200 FPS.
Challenge: This approach reduces the number of available viewpoints at any single timestamp, leading to sparse-view reconstruction issues and visible "floater" artifacts.

2. Artifact-Fix Video Diffusion Model

To address the artifacts caused by sparse views and temporal inconsistency, the authors propose a generative refinement module.

Motivation: While image diffusion models can fix spatial artifacts, they process frames independently, leading to temporal inconsistency in 4D reconstruction.
Architecture: The model is built upon the Wan2.1 video diffusion model. It utilizes a Video-VAE to compress rendered videos into latent space. The model takes the noisy, artifact-ridden video (rendered from the initial sparse 4D Gaussian model) as a condition and generates a clean, temporally coherent video.
Training Strategy:
- Data Curation: A dataset of 750 pairs is constructed by temporally sub-sampling existing 4D sequences (DNA-Rendering) to simulate asynchronous capture and large motion. A 4D Gaussian Splatting (GS4D) model is trained on these sparse inputs to generate "noisy" renderings, which are paired with the original ground-truth videos.
- Fine-tuning: The model is fine-tuned using LoRA (Low-Rank Adaptation) on the DiT (Diffusion Transformer) blocks while freezing the encoder and decoder. The training objective minimizes the difference between the generated latent and the target latent.
Integration: The refined video is used to supervise the 4D Gaussian model through a combined loss function involving $L_1$ distance and perceptual loss (LPIPS), distilling the video diffusion prior into the 4D reconstruction.

3. 4D Reconstruction Pipeline

The overall pipeline (illustrated in Fig. 2) proceeds as follows:

Initialization: An initial 4D Gaussian Splatting (GS4D) model is reconstructed from the asynchronously captured multi-view videos.
Rendering: High-frame-rate videos are rendered from this initial model, which inherently contain artifacts due to viewpoint sparsity.
Refinement: The artifact-fix video diffusion model processes these rendered videos to produce clean, temporally consistent outputs.
Optimization: The 4D Gaussian model is further optimized using the refined videos as supervision, improving both spatial details and temporal consistency.

Key Contributions

Hardware Solution: A low-cost asynchronous capture system that increases effective frame rates by staggering camera start times, enabling 100–200 FPS reconstruction using standard 25–30 FPS cameras.
Software Solution: A novel artifact-fix video diffusion model specifically trained to handle sparse-view 4D reconstruction. Unlike image-based methods, it leverages spatiotemporal priors to remove "floater" artifacts while maintaining temporal consistency.
Dataset: The creation of the first real-world dataset containing 12 sequences of asynchronous multi-view videos capturing fast and complex dynamic scenes, validating the practical feasibility of the approach.

Experimental Results

The method was evaluated on synthetic datasets (DNA-Rendering, Neural3DV) and a custom real-world capture setup.

Quantitative Performance: On the DNA-Rendering dataset, the proposed method achieved a PSNR of 26.76, SSIM of 0.845, and LPIPS of 0.293, outperforming state-of-the-art baselines including K-Planes, 4DGS, and GS4D. Similar improvements were observed on the Neural3DV dataset (PSNR: 33.48).
Ablation Studies:
- Asynchronous vs. Synchronous: Asynchronous capture significantly improved reconstruction quality for large motions compared to synchronous capture, even without the diffusion model.
- Video vs. Image Diffusion: Using a video diffusion model resulted in superior temporal consistency compared to image diffusion models, which introduced texture variations between frames.
- Artifact Fixing: The inclusion of the artifact-fix model significantly reduced LPIPS and improved visual fidelity, particularly in regions with complex textures and non-linear motion (e.g., skirts, waving arms).
Real-World Validation: In real-world captures of dancing and sports, the method successfully reconstructed high-speed motions that were unresolvable by synchronous capture or standard interpolation methods.

Significance and Claims

The paper claims that 4DSloMo provides a robust joint hardware-software solution for high-speed 4D reconstruction. By decoupling the capture frame rate from the camera's native frame rate through asynchronous triggering, the method overcomes the limitations of existing 4D capture arrays without requiring expensive high-speed hardware.

Furthermore, the authors emphasize that their video-diffusion-based artifact-fixing approach resolves the temporal inconsistency issues inherent in 4D reconstruction, a problem that image-diffusion-based refinement methods fail to address. The work demonstrates that it is possible to recover intermediate frames of high-speed motion using regular video cameras, provided that the capture strategy and reconstruction pipeline are co-designed to handle the resulting sparsity and motion complexity. The release of the first asynchronous multi-view dataset is intended to facilitate further research in this domain.