💻 computer science

ReplicateAnyScene: Zero-Shot Video-to-3D Composition via Textual-Visual-Spatial Alignment

The paper introduces ReplicateAnyScene, a zero-shot framework that transforms casually captured videos into compositional 3D scenes by aligning textual, visual, and spatial priors from foundation models, while also proposing the C3DR benchmark to evaluate reconstruction quality.

Original authors: Mingyu Dong, Chong Xia, Mingyuan Jia, Weichen Lyu, Long Xu, Zheng Zhu, Yueqi Duan

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mingyu Dong, Chong Xia, Mingyuan Jia, Weichen Lyu, Long Xu, Zheng Zhu, Yueqi Duan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a messy room, glance at it for a few seconds, and then instantly close your eyes. In your mind, you don't just see a blurry mess; you see a chair, a table, a bag, and a lamp. You know exactly where the chair is sitting, how heavy it feels, and that the lamp is resting on the table, not floating in mid-air. You could even build a perfect 3D model of that room in your head, piece by piece.

For a long time, computers have been terrible at this. They could either take a photo and make a flat 2D picture look 3D (like a hologram), or they could build a 3D model if you gave them a million photos and a lot of human help. But they couldn't just "watch" a video, understand what the objects are, and build a realistic, separate 3D model of every single item in the scene automatically.

Enter ReplicateAnyScene. Think of this as a super-smart, fully automated digital architect that can watch a casual video of a room and instantly build a perfect, interactive 3D version of it, object by object.

Here is how it works, broken down into five simple steps using a creative analogy:

The Analogy: Building a LEGO City from a Video

Imagine you are trying to rebuild a city using only a shaky video taken by a tourist. You have a team of experts (the AI models) helping you. Here is your five-step construction crew:

1. The Scout (Progressive Object Discovery)

The Problem: If you show the whole video to a computer at once, it gets overwhelmed and forgets things.
The Solution: The computer acts like a scout. Instead of watching the whole movie, it picks the best 20 frames (like the most interesting snapshots). It asks a "smart librarian" (a Vision-Language Model): "What objects do you see here?"
The Magic: If the librarian says "sofa" in one frame and "couch" in another, the computer knows they are the same thing. It creates a clean, non-repeating shopping list of everything in the room: Chair, Table, Bag, TV.

2. The Detective (Spatial-Guided Visual Deduplication)

The Problem: In a video, a chair might be hidden behind a table for a second, then reappear. A dumb computer might think, "Oh, that's a new chair!" and create two chairs instead of one.
The Solution: The computer acts like a detective who knows that real objects don't teleport. It takes the 2D pictures of the chair and lifts them into 3D space. If two "chairs" occupy the same physical space in 3D, the detective merges them into one. It ensures that every object in the final model is unique and real.

3. The Photographer (Optimal-View 3D Asset Generation)

The Problem: To build a 3D model of a chair, you need a good photo. But if you pick a photo where the chair is tiny or blocked, the 3D model will look weird.
The Solution: The computer acts like a professional photographer. It looks at all the frames where the chair appears and asks: "Which angle shows the most of the chair's surface?" It picks that perfect shot and uses a powerful AI artist to generate a high-quality 3D model of just that chair, complete with textures and details.

4. The Glue (Iterative Visual-Spatial Alignment)

The Problem: Now you have a 3D chair and a 3D table, but they are floating in empty space. The computer needs to put them in the right spot.
The Solution: This is the "glue" stage. The computer tries to fit the 3D chair into the video. It projects the 3D chair onto the video frame and checks: "Does the edge of my 3D chair match the edge of the real chair in the video?" If it's off, it nudges the chair slightly and tries again. It does this over and over (like a puzzle solver) until the 3D model fits the video perfectly.

5. The Physicist (Semantic-Aware Scene Refinement)

The Problem: Sometimes, the computer gets the physics wrong. It might put a chair floating in the air, or a lamp hanging from the ceiling upside down.
The Solution: The computer calls in a "physicist" (another smart AI). It asks: "Does this make sense?"

If the chair is floating, the physicist says, "Gravity! Put it on the floor."
If the lamp is upside down, it says, "Lamps sit on tables, they don't hang from the ceiling."
It makes tiny adjustments to ensure the scene looks physically real and logical.

Why is this a big deal?

Before this, building 3D worlds for robots, video games, or virtual reality required:

Manual labor: Humans had to draw every object.
Special equipment: You needed expensive depth cameras.
Simple scenes: It only worked on boring, empty rooms.

ReplicateAnyScene changes the game. It is Zero-Shot, meaning it doesn't need to be trained on specific rooms; it can handle any room you show it. It is Fully Automated, meaning you just press "play" on a video, and it does the rest.

The Result

The researchers also built a new "test track" called C3DR to grade these 3D models. Their system scored higher than any previous method, creating scenes that are not only visually beautiful but also physically correct.

In short: This technology turns a simple video into a fully interactive, 3D world where every object is real, placed correctly, and ready for robots or gamers to explore. It's like giving a computer the human ability to "see" and "understand" the 3D world around us.

1. Problem Statement

Compositional 3D Reconstruction aims to extract complete object geometries from video and assemble them into structured, physically plausible 3D scenes, distinct from holistic reconstruction methods (like NeRF or 3DGS) that treat scenes as a single entity. This capability is crucial for Spatial Intelligence and Embodied AI.

Current Limitations:
Existing methods struggle with real-world deployment due to:

Manual Dependency: Heavy reliance on manual object prompting or annotations.
Auxiliary Input Requirements: Dependence on precise depth maps, semantic masks, or normal priors.
Training Biases: Restricted to simplistic scenes, failing in complex, cluttered environments.
Cross-Modal Fragmentation: Naive integration of isolated Vision Foundation Models (VFMs) leads to scale ambiguities, identity mismatches, and disjointed spatial layouts (e.g., floating objects, collisions).
Lack of Evaluation: No standardized benchmark exists to evaluate compositional reconstruction across textual, visual, and spatial dimensions.

2. Methodology: ReplicateAnyScene

The authors propose ReplicateAnyScene, a fully automated, zero-shot framework that transforms casually captured monocular videos into compositional 3D scenes. The core innovation is a five-stage cascade pipeline that progressively aligns textual, visual, and spatial priors extracted from VFMs.

Stage 1: Progressive Object Discovery

Goal: Generate a non-redundant, open-vocabulary list of object categories in the video.
Mechanism: Instead of feeding the entire video to a Vision-Language Model (VLM) (which causes context overflow), the system uses a space-aware frame sampling algorithm to select $N_k=20$ frames.
Process: A VLM iteratively scans these frames, identifying only novel objects distinct from a dynamic category registry. This deduplicates synonyms (e.g., "sofa" vs. "couch") and builds a concise category list.

Stage 2: Spatial-Guided Visual Deduplication

Goal: Resolve 2D tracking fragmentation and ensure consistent instance identities across the video.
Problem: 2D trackers (like SAM3) often split a single physical object into multiple mask sequences due to occlusion or re-entry.
Mechanism:
1. Lift 2D masks into 3D space using depth maps and camera poses (from VGGT).
2. Aggregate masks into instance-level point clouds.
3. Calculate geometric overlap between point clouds of the same semantic category using nearest-neighbor distances.
4. If overlap exceeds a threshold, merge fragments using a Union-Find data structure to assign a global, consistent ID.

Stage 3: Optimal-View 3D Asset Generation

Goal: Generate high-fidelity 3D meshes for each unique instance.
Mechanism: Uses SAM3D to synthesize textured meshes from 2D masks and point maps.
Optimization: Instead of picking the frame with the largest 2D mask (which may be a close-up with poor visibility), the system selects the view that maximizes the reconstructed 3D surface area. This ensures the generation model receives the most comprehensive geometric information.

Stage 4: Iterative Visual-Spatial Alignment

Goal: Precisely align generated 3D assets with the global scene structure.
Problem: Initial generation parameters often suffer from spatial misalignment.
Mechanism: An iterative Render-Match-Optimize cycle inspired by ICP but driven by visual correspondences:
1. Render: Render the 3D asset using current pose parameters.
2. Match: Use MASt3R to establish dense 2D correspondences between the real video frame and the rendered image.
3. Lift: Back-project matches into 3D world space.
4. Optimize: Solve for the optimal similarity transformation (scale, rotation, translation) using the Umeyama algorithm to minimize geometric error.
5. Selection: Choose the iteration with the highest Intersection-over-Union (IoU) between rendered and real masks.

Stage 5: Semantic-Aware Scene Refinement

Goal: Enforce physical plausibility and correct spatial anomalies (e.g., floating, tilting).
Mechanism: A VLM infers spatial relationships (e.g., "supported by," "attached to").
Process: Based on these semantic priors, predefined rules are applied to the 3D assets:
- Align vertical axes with gravity.
- Adjust translation to ensure contact with supporting planes (floors/walls).
- This step eliminates residual artifacts that purely geometric optimization misses.

3. Key Contributions

ReplicateAnyScene Framework: A fully automated, zero-shot pipeline for compositional 3D reconstruction from casual videos, eliminating the need for manual prompts or auxiliary depth inputs.
Multimodal Alignment Strategy: A novel five-stage cascade that structurally aligns textual, visual, and spatial priors from diverse VFMs, solving issues of scale ambiguity and identity fragmentation.
C3DR Benchmark: The introduction of the Compositional 3D Reconstruction Benchmark, a dataset of 50 complex scenes with ground truth constructed via manual annotation and high-fidelity asset generation. It evaluates performance across:
- Textual: Category recall and instance fidelity.
- Visual: Rendering quality (PSNR, SSIM, LPIPS).
- Geometric: Spatial accuracy (Chamfer Distance, F-Score).
State-of-the-Art Performance: The method outperforms existing baselines (MetaScenes, SimRecon) significantly in all evaluation metrics.

4. Experimental Results

Quantitative Performance: On the C3DR benchmark, ReplicateAnyScene achieved:
- Textual: 91.43% Recall and 90.86% F1-score (vs. 64.29% and 75.10% for MetaScenes).
- Visual: Highest PSNR (19.45), SSIM (0.854), and MUSIQ (71.03).
- Geometric: Superior Normal Consistency (NC) and F-Score compared to baselines.
Qualitative Improvements:
- Deduplication: Successfully merged fragmented object tracks, preventing duplicate objects.
- Alignment: Eliminated floating artifacts and scale distortions common in ICP-only or unaligned methods.
- Refinement: Corrected unnatural tilting and ensured objects rested correctly on surfaces.
Ablation Studies: Confirmed that removing any of the five stages (especially Deduplication and Semantic Refinement) leads to significant degradation in object fidelity and physical plausibility.

5. Significance and Impact

Advancement in Spatial Intelligence: Provides a scalable solution for generating interactive 3D environments from unstructured video, a critical step for Embodied AI agents that require object-level manipulation.
Zero-Shot Generalization: Demonstrates that foundation models can be synergized to handle complex, open-vocabulary scenes without task-specific training data.
Standardization: The C3DR benchmark fills a critical gap in the field, providing a rigorous standard for evaluating future compositional reconstruction methods.
Future Applications: The generated datasets can support downstream tasks such as scene editing agents, video diffusion conditioned on 3D assets, and robotic navigation training.

Limitations: The current framework is optimized for indoor environments and struggles with complex natural scenes containing irregular geometries or unbounded spaces, which remain open challenges.