💻 computer science

AnyRecon: Arbitrary-View 3D Reconstruction with Video Diffusion Model

AnyRecon is a scalable framework that enables robust 3D reconstruction from arbitrary, unordered sparse inputs by leveraging a video diffusion model enhanced with a persistent global scene memory, geometry-aware conditioning, and efficiency optimizations to ensure geometric consistency across large viewpoint gaps.

Original authors: Yutian Chen, Shi Guo, Renbiao Jin, Tianshuo Yang, Xin Cai, Yawen Luo, Mingxin Yang, Mulin Yu, Linning Xu, Tianfan Xue

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yutian Chen, Shi Guo, Renbiao Jin, Tianshuo Yang, Xin Cai, Yawen Luo, Mingxin Yang, Mulin Yu, Linning Xu, Tianfan Xue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a video of a room taken from just a few random angles—maybe you walked in, snapped a photo, turned around, snapped another, and then stopped. You want to turn this into a full, explorable 3D world where you can walk around, look behind the sofa, and see what's on the other side of the room.

Doing this is incredibly hard for computers. Usually, they need hundreds of photos taken in a perfect grid to build a 3D model. If the photos are sparse (few in number) and messy (taken from random spots), the computer gets confused and the 3D model falls apart.

Enter AnyRecon. Think of it as a "3D Time Traveler" that can take your messy, sparse photos and fill in the gaps to create a perfect, walkable 3D world. Here is how it works, using some everyday analogies:

1. The Problem: The "Blind Artist"

Most current AI tools trying to do this are like a painter who is only allowed to look at two reference photos (the first and the last one you took). They try to guess what's in the middle.

The Issue: If the room is big or complex, guessing based on just two photos leads to a blurry, distorted mess. The AI forgets what the table looked like in the middle of the room because it can't "remember" the specific details from the photos you actually took.

2. The Solution: The "Super Librarian" (Global Scene Memory)

AnyRecon changes the game. Instead of just looking at two photos, it acts like a Super Librarian.

How it works: It takes all the photos you have (even if they are in random order) and puts them in a special "Global Memory Cache."
The Analogy: Imagine you are trying to describe a house to a friend. Instead of just showing them the front door and the back door, you give them a photo album of every room. AnyRecon keeps this album open and constantly refers to it. No matter where you want to "look" in the 3D world, the AI can instantly pull up the exact photo from your album that helps it understand that specific angle.

3. The Secret Sauce: No "Fast-Forwarding" (No Temporal Compression)

Video AI usually works by "compressing" time, kind of like skipping frames in a movie to make it load faster. It assumes the camera moves smoothly from one frame to the next.

The Issue: In real life, you might take a photo, walk 10 feet, turn 90 degrees, and take another. That's a huge jump! If the AI "skips" the time between them, it loses the sharp details.
The Fix: AnyRecon refuses to skip frames. It treats every single photo as a high-definition, standalone piece of the puzzle. It doesn't try to "smooth over" the gaps; instead, it respects the sharp, distinct details of every photo you took, ensuring the 3D model stays crisp and doesn't turn into a blurry smear.

4. The Loop: The "Architect and the Builder"

This is the most clever part. AnyRecon doesn't just generate images; it builds a 3D structure while it generates.

The Process:
1. The Architect (3D Memory): It builds a rough 3D skeleton of the scene based on your photos.
2. The Builder (Diffusion Model): It uses that skeleton to paint new, realistic views of the room.
3. The Update: As soon as it paints a new view, it scans that new image, finds the 3D details, and adds them back into the Architect's skeleton.
The Result: It's a self-improving loop. The more it generates, the better its 3D map becomes, which helps it generate even better images next time. It's like a sculptor who keeps adding clay and then stepping back to refine the shape, over and over again.

5. The Smart Selector: "The Right Reference"

When the AI needs to generate a new view, it doesn't just grab random photos from your album. It uses Geometry-Driven Retrieval.

The Analogy: Imagine you are trying to draw a tree from a new angle. You wouldn't look at a photo of the tree's roots or a photo of the sky above it. You would look for the photo that shows the tree from a similar angle.
How it works: AnyRecon calculates exactly which of your original photos provides the best "geometric support" for the new view. It ignores photos that are blocked by walls (occluded) or irrelevant, ensuring it only uses the most helpful references.

Why This Matters

Before AnyRecon, turning a casual video or a few random photos into a high-quality 3D world was slow, blurry, or impossible for large scenes.

Speed: It's incredibly fast (about 20 times faster than previous methods) because it uses a "distillation" technique (like a student learning from a master teacher to solve problems in 4 steps instead of 50).
Versatility: It works whether you have 3 photos or 200, whether they are close together or far apart.

In short: AnyRecon is a tool that takes your messy, sparse snapshots and, by acting as a super-organized librarian and a self-correcting architect, turns them into a smooth, high-definition 3D world you can explore.

1. Problem Statement

Sparse-view 3D reconstruction from casual captures (e.g., handheld videos or internet clips) is a fundamental challenge in computer vision. Existing methods face two primary limitations:

Non-Generative Methods: Traditional approaches (e.g., NeRF, 3D Gaussian Splatting) rely on dense, controlled multi-view inputs. They struggle with large viewpoint gaps, irregular inputs, and occlusions, often failing to synthesize plausible details in unobserved regions.
Existing Diffusion-Based Methods: While recent works use diffusion models to synthesize novel views, they typically condition on only one or two frames (e.g., the first and last views). This restricts geometric consistency, limits scalability to large scenes, and fails to handle arbitrary, unordered input sequences effectively. Furthermore, standard video diffusion models rely on temporal compression, which breaks frame-level correspondence when viewpoint gaps are large.

Goal: To enable high-quality, scalable 3D reconstruction from arbitrary, unordered, and sparse inputs while maintaining explicit geometric control and computational efficiency.

2. Methodology: AnyRecon Framework

AnyRecon proposes a closed-loop framework that alternates between diffusion-based novel view synthesis and geometry refinement. The pipeline consists of three main stages:

A. Unordered Contextual Video Diffusion

To handle arbitrary inputs, the authors modify the standard video diffusion architecture:

Global Scene Memory: Instead of treating inputs as a strictly temporal sequence, the model prepends retrieved reference views ( $I_{sel}$ ) to the generation sequence. These views act as a persistent Key-Value (KV) memory cache within the Video Diffusion Transformer. This allows the model to condition on an arbitrary number of unordered frames, enabling long-range spatial reasoning independent of capture order.
Non-Compressive Latent Encoding: Standard video diffusion models use temporal compression (e.g., 3D-VAEs) which assumes temporal smoothness. AnyRecon removes this compression, using a frame-wise 2D VAE instead. This preserves a one-to-one mapping between latent tokens and pixel coordinates, preventing feature entanglement across large viewpoint gaps and ensuring precise geometric alignment.
Geometric Conditioning: The model receives explicit geometric guidance by concatenating rendered point-cloud observations ( $I_{render}$ ) and visibility masks ( $M_t$ ) derived from the current 3D geometry memory along with the noisy latents.

B. Geometry-Aware Conditioning Strategy

To support large-scale, segment-by-segment reconstruction, the generation process is tightly coupled with an explicit 3D representation:

3D Geometry Memory Update ( $M_{geo}$ ): The system maintains an incrementally updated point cloud. After synthesizing a segment of the trajectory, the new views are back-projected using a feed-forward point map estimator (e.g., $\pi_3$ ) to update the global point cloud. This updated geometry anchors subsequent generation steps, preventing geometric drift.
Geometry-Driven View Selection: Instead of selecting views based on image similarity or Field-of-View (FOV), the system performs geometry-driven retrieval. It renders the current $M_{geo}$ from the target viewpoint to generate a visibility index map. It then selects the top- $k$ capture views that contribute the most visible 3D points to the target, filtering out occluded or irrelevant views.

C. Efficient Inference Architecture

To handle the computational cost of long sequences and high-resolution outputs:

Context-Window Sparse Attention: To mitigate the $O(L^2)$ complexity of full attention with long sequences, the model employs sparse attention. Each frame attends only to a local temporal window and a selectively retrieved subset of geometry-aligned reference views.
4-Step Diffusion Distillation: The pre-trained video diffusion model (Wan2.1) is distilled into a student network capable of high-quality generation in just 4 steps using Distribution Matching Distillation (DMD). This reduces inference time significantly without substantial quality loss.

3. Key Contributions

Flexible Sparse-View Framework: A video-diffusion-based approach that supports arbitrary and unordered conditioning views, overcoming the limitations of fixed-length or sequential inputs.
Geometry-Aware Conditioning: A novel strategy that couples generation and reconstruction via an explicit 3D Geometry Memory and geometry-driven view retrieval, ensuring spatially grounded diffusion guidance.
Efficient Architecture: A design that removes temporal compression to preserve geometric fidelity and utilizes sparse attention combined with 4-step distillation to achieve a 20x speedup in inference while maintaining high quality.

4. Experimental Results

The authors evaluated AnyRecon on the DL3DV-10K and Tanks and Temples datasets, comparing it against state-of-the-art baselines (Difix3D+, ViewCrafter, Uni3C).

Quantitative Performance:
- Interpolation: Achieved 20.95 PSNR and 0.656 SSIM on DL3DV, significantly outperforming ViewCrafter (15.86 PSNR) and Uni3C (16.33 PSNR).
- Extrapolation: Achieved 21.16 PSNR, demonstrating superior ability to hallucinate coherent content beyond observed ranges.
- Efficiency: Reduced inference time to 105 seconds per 40-frame sequence, compared to 170s (ViewCrafter), 340s (Uni3C), and 1200s (Difix3D+).
Qualitative Performance:
- Robustness: Successfully handles large viewpoint gaps and irregular inputs where other methods fail to maintain structural integrity or suffer from color shifts.
- Detail Preservation: The non-compressive encoding and global memory allow for the recovery of fine-grained textures (e.g., tableware, metal grids) that are often blurred or lost in compressed latent spaces.
- Consistency: The closed-loop geometry update prevents the "drift" seen in iterative methods, ensuring consistent scene structure across long trajectories.

5. Significance

AnyRecon represents a significant leap forward in casual 3D reconstruction. By bridging the gap between generative priors (diffusion) and explicit geometric control, it solves the scalability and consistency issues that have plagued previous sparse-view methods.

Practical Application: It enables the conversion of everyday, sparse video captures (like smartphone footage) into explorable, high-fidelity 3D assets, which is crucial for AR/VR, digital twins, and visual effects.
Architectural Insight: The paper demonstrates that removing temporal compression and utilizing explicit geometric memory are critical for handling large viewpoint gaps, offering a new direction for future video diffusion models in 3D tasks.