One2Scene: Geometric Consistent Explorable 3D Scene Generation from a Single Image

One2Scene is a novel framework that generates geometrically consistent, explorable 3D scenes from a single image by decomposing the task into panorama generation, 3D scaffold construction via multi-view stereo matching on sparse anchor views, and novel view synthesis, thereby overcoming the severe distortions and artifacts common in existing methods during large camera motions.

The Gist

Imagine you have a single photograph of a room. You want to turn that flat picture into a fully explorable 3D world where you can walk around, look behind the sofa, and peek out the window.

This is a notoriously difficult problem for computers. If you just try to "guess" what's behind the camera, the computer often gets confused. It might stretch the walls like taffy, make the floor disappear, or create a hallway that leads nowhere. This is what happens with current technology: it's like trying to build a house by guessing the blueprint from a single photo of the front door.

The paper "One2Scene" introduces a new method that solves this by breaking the impossible task into three manageable steps, using a clever "construction crew" approach.

Here is how it works, explained with simple analogies:

Step 1: The "360-Degree Panoramic Map" (The Panorama Generator)

The Problem: A single photo only shows you what's in front of you. It's like looking through a keyhole.
The Solution: The system first uses an AI artist to imagine the rest of the room. It takes your single photo and expands it into a 360-degree panoramic map (like a Google Street View sphere).

• Analogy: Imagine you are standing in the middle of a room. You can only see the wall in front of you. One2Scene acts like a magical painter who instantly paints the other three walls, the ceiling, and the floor around you, giving you a complete "bubble" of the world.

Step 2: The "Scaffolding Crew" (The 3D Geometric Scaffold)

The Problem: Even though the system now has a 360-degree picture, it's still just a flat painting wrapped around a sphere. It doesn't know how far away the walls are. If you tried to walk through it, you might walk right into a painted wall.
The Solution: This is the core innovation. The system takes that flat 360-degree map and cuts it into six square "anchor views" (like the six faces of a dice). It then uses a special "feed-forward" network (a fast, one-shot calculator) to turn these flat squares into a 3D geometric scaffold.

• Analogy: Think of building a house. Before you put up the pretty wallpaper or paint, you need a sturdy wooden frame (scaffolding).
  • Old methods tried to guess the shape of the house while painting the walls, often resulting in crooked, wobbly structures.
  • One2Scene builds a perfect, rigid wooden frame first. It calculates exactly where every wall, floor, and ceiling is in 3D space. It does this by treating the six "dice faces" like a puzzle, using math to figure out the depth and distance between them.

Step 3: The "Interior Designer" (The Novel View Synthesis)

The Problem: Now you have a sturdy 3D frame, but it's empty. You need to fill it with realistic details (textures, lighting, objects) that look good from any angle you choose to walk to.
The Solution: The system uses the 3D scaffold as a "guide rail." It tells the AI generator: "Hey, we know exactly where the wall is. Now, please paint a realistic image of that wall from this new angle."

• Analogy: Imagine you are a director filming a movie. You have a rigid set (the scaffold) built by the construction crew. Now, the camera can move anywhere—up, down, left, right—because the set is solid. The "Interior Designer" AI just needs to paint the walls and furniture based on where the camera is pointing. Because the "scaffolding" is already there, the AI never gets confused about where the floor is or how big the room is.

Why is this a big deal?

1. No More "Wobbly" Worlds: Previous methods often created "hallucinations" where objects stretched or disappeared when you moved the camera. Because One2Scene builds a solid 3D skeleton first, the world stays stable and consistent, no matter how far you walk.
2. Speed: It doesn't need to spend hours optimizing every single scene. It builds the scaffold in 0.5 seconds.
3. Exploration: You can actually "walk" through the generated scene. If you turn around 180 degrees, the system knows exactly what should be there because the 3D scaffold told it.

Summary

Think of One2Scene as a three-step construction project:

1. The Artist: Draws a complete 360-degree picture of the world.
2. The Engineer: Builds a perfect, invisible 3D wireframe (scaffold) inside that picture so the geometry is correct.
3. The Painter: Uses that wireframe to paint realistic, high-quality views from any angle you want.

By separating the "geometry" (the shape) from the "appearance" (the look), One2Scene creates 3D worlds that are not only beautiful but also physically consistent, allowing for true, immersive exploration from a single photo.

Technical Summary

1. Problem Statement

The paper addresses the challenge of generating fully explorable 3D scenes from a single input image. While existing methods have made progress in 3D generation, they face critical limitations when dealing with large viewpoint changes and long-term exploration:

• Geometric Distortion: Methods often suffer from severe geometric inconsistencies, artifacts, and "semantic drift" when the camera moves far from the original perspective.
• Scale Ambiguity: Single-image inputs often lead to physically implausible results (e.g., objects changing size or cameras passing through walls) due to a lack of global scale constraints.
• Error Accumulation: Iterative or sequential approaches (like inpainting or video diffusion) tend to accumulate errors over time, leading to geometric collapse in loop-closure scenarios.
• Limited Exploration: Many current methods produce 360° panoramas that cannot be navigated freely or fail to synthesize high-quality novel views from arbitrary, unseen camera poses.

2. Methodology: The One2Scene Framework

One2Scene decomposes the ill-posed single-image-to-scene problem into three tractable sub-tasks, forming a three-stage pipeline:

Stage 1: Panorama Generation (Anchor View Initialization)

• Goal: Overcome the information deficit of a single image by expanding the visual field.
• Approach: The input image is processed by a specialized image-to-panorama generator (specifically Hunyuan-Pano-DiT) to create a 360° panoramic representation.
• Rationale: A panoramic view provides a comprehensive field of view with more visual cues, making the subsequent geometric reconstruction more robust than attempting direct arbitrary view synthesis.

Stage 2: Feed-Forward 3D Geometric Scaffold Construction

• Goal: Lift the 2D panorama into an explicit, geometrically consistent 3D structure.
• Key Innovation (Anchor View Projection): Instead of treating the panorama as a single image, the system projects the 360° panorama into six perspective cubemap views (anchor views). To facilitate matching, the Field of View (FoV) of each cube face is expanded to 95°, creating a 2.5° overlap between adjacent views.
• Key Innovation (Bidirectional Fusion Module): The authors adapt the VGGT (Visual Geometry Grounded Transformer) architecture. Since the overlap between anchor views is extremely sparse (2.5°), standard multi-view stereo models fail. One2Scene introduces a Bidirectional Fusion Module that:
  1. Transforms features from the six cubemap views into a unified equirectangular latent space (Cube-to-Equirectangular, C2E).
  2. Fuses these features using a convolutional layer to enforce global consistency.
  3. Projects the fused features back to the cubemap space (Equirectangular-to-Cube, E2C) and merges them with original features via residual connections.
  • Result: This ensures geometric consistency across view boundaries while preserving view-specific details.
• Output: A 3D Gaussian Splatting (3DGS) model predicts Gaussian parameters ($\mu, \alpha, \Sigma, c$) for every pixel, creating an explicit 3D geometric scaffold in 0.5 seconds.

Stage 3: Scaffold-Guided Novel View Synthesis

• Goal: Generate photorealistic, consistent novel views from arbitrary camera poses using the 3D scaffold as a prior.
• Key Innovation (Dual-LoRA Training): The synthesis network (based on SEVA) is conditioned on two distinct types of information:
  1. High-quality Anchor Views: Pristine appearance but geometrically ambiguous.
  2. Rendered Views from Scaffold: Geometrically accurate but potentially containing artifacts (e.g., black holes due to occlusion).
  • Instead of simple concatenation, the model uses Dual-LoRA modules to process these heterogeneous signals independently before fusing them via a 3D Attention Mechanism. This allows the model to leverage the geometric priors of the scaffold without being corrupted by its rendering artifacts.
• Memory Condition: To ensure temporal consistency during continuous exploration, a "memory bank" of previously generated frames (closest in camera pose) is used as an additional condition to guide the synthesis of occluded regions.

3. Key Contributions

1. Three-Stage Decomposition: A novel framework that systematically breaks down the single-image-to-3D problem into panorama generation, explicit scaffold construction, and guided synthesis.
2. Feed-Forward 3DGS with Bidirectional Fusion: A highly efficient (0.5s) 3D reconstruction model that reformulates monocular panoramic depth estimation as a sparse multi-view stereo problem, utilizing a bidirectional fusion module to handle sparse overlaps.
3. Dual-LoRA Synthesis Strategy: A training mechanism that effectively fuses high-quality appearance data with geometrically rich (but noisy) scaffold renderings, resolving the scale ambiguity and geometric drift common in single-image generation.
4. State-of-the-Art Performance: The method achieves superior results in geometric stability, scale consistency, and photorealism compared to existing SOTA methods.

4. Experimental Results

The authors evaluated One2Scene on diverse datasets (indoor/outdoor, real/stylized) against methods like DreamScene360, WonderJourney, VMem, and SEVA.

• Quantitative Performance:
  • Visual Quality: Achieved the best NIQE (4.43) and Q-Align (4.13) scores, indicating superior naturalness and perceptual quality.
  • Semantic Consistency: Achieved the highest CLIP-I score (89.95), showing better preservation of the original scene's content and style.
  • Geometric Stability: Significantly outperformed competitors in Camera Motion Consistency (CamMC) (0.389 vs. 0.998 for VMem) and Translation/Rotational Errors, demonstrating robust long-term exploration without geometric collapse.
• Efficiency: The 3D scaffold is constructed in 0.5 seconds on an H20 GPU, which is 5.6× faster than dense-view baselines like AnySplat.
• Depth Estimation: The underlying depth estimation model achieved state-of-the-art results on Stanford2D3D (zero-shot) and Matterport3D (fine-tuned), outperforming specialized panoramic depth estimators.

5. Significance

One2Scene represents a significant leap forward in 3D content creation by solving the "explorability" bottleneck.

• Robustness: It effectively prevents the error accumulation and geometric drift that plague iterative and video-diffusion-based methods, enabling stable navigation through large scenes.
• Generalization: By leveraging a 3D geometric scaffold, it generalizes well to both indoor and outdoor environments and various visual styles (realistic and stylized).
• Practicality: The feed-forward nature of the scaffold generation (sub-second inference) makes it viable for real-time applications in VR/AR, gaming, and virtual production, where users require immediate, high-fidelity, and navigable 3D environments from a single image.

In summary, One2Scene shifts the paradigm from "generating views sequentially" to "constructing a geometric prior first," ensuring that the resulting 3D scenes are not only visually stunning but also geometrically coherent and truly explorable.