ZeroScene: A Zero-Shot Framework for 3D Scene Generation from a Single Image and Controllable Texture Editing

ZeroScene is a novel zero-shot framework that leverages large vision models to reconstruct coherent 3D scenes from a single image and enables controllable, multi-view consistent texture editing through joint optimization of 3D/2D losses and mask-guided diffusion strategies.

The Gist

Imagine you have a single photograph of a messy room: a teddy bear sitting on a wooden table, a cup nearby, and a poster on the wall. Now, imagine you want to turn that flat, 2D photo into a fully playable 3D video game level where you can walk around, pick up the bear, and even change the cup's design to look like a golden trophy.

That is exactly what ZeroScene does, but it does it with a magic trick called "Zero-Shot" learning. It doesn't need to be taught with thousands of examples; it just looks at your picture and figures it out on the fly.

Here is how the system works, broken down into simple steps with some creative analogies:

1. The "Cut and Paste" Detective (Scene Reconstruction)

When you show ZeroScene a photo, it doesn't just see a flat image. It acts like a super-smart detective that can see through the clutter.

• The Problem: In a photo, objects often hide behind each other (occlusion). If a teddy bear is half-hidden by a table, a normal computer might think the bear is broken or missing a leg.
• The ZeroScene Solution: It first "cuts out" every object it sees (the bear, the cup, the table). Then, it uses a powerful AI "imagination engine" (a Large Vision Model) to fill in the missing parts. It's like looking at a torn puzzle piece and instantly knowing what the rest of the picture looks like, so it can "heal" the image of the bear before turning it into 3D.
• The Background: Most 3D tools ignore the background (the walls and floor) because they are hard to model. ZeroScene treats the background as a solid foundation. It removes the objects, figures out the shape of the room, and then puts the objects back in their exact correct spots.

2. The "3D Jigsaw Puzzle" (Layout Optimization)

Once the AI has created 3D models of the bear, the cup, and the table, they might be floating in space or sized incorrectly. They need to be arranged just like they were in the original photo.

• The Analogy: Imagine you have a bunch of 3D printed toys and you want to arrange them on a table to match a photo. You might guess the position, but it's hard to get the depth right.
• The ZeroScene Solution: It uses a "double-check" system. It projects the 3D models back onto a 2D screen and compares them to the original photo. If the bear looks too big or too far away, the system nudges it until the 3D scene perfectly matches the 2D photo from every angle. It's like tuning a radio until the static disappears and the music is crystal clear.

3. The "Magic Paintbrush" (Texture Editing)

Now that you have a 3D scene, what if you want to change the bear's fur to look like a shiny metal robot, or turn the wooden table into a glass surface?

• The Problem: If you just paint a 3D object, the texture often looks weird when you walk around it (like a sticker peeling off). It might look great from the front but blurry from the side.
• The ZeroScene Solution: It uses a "Mask-Guided Progressive" strategy. Think of this as painting a mural on a curved wall. Instead of painting the whole wall at once, you paint a small section, then move to the next, making sure the new paint blends perfectly with the old paint.
• The Result: You can type "Make the cup look like a pink heart," and ZeroScene will repaint the cup from every single angle simultaneously, ensuring the heart pattern wraps around the cup smoothly without any seams or glitches.

4. The "Realism Booster" (PBR Materials)

Finally, to make the scene look like a movie and not a cartoon, ZeroScene adds Physically Based Rendering (PBR).

• The Analogy: Regular 3D models are like paper cutouts; they look flat even when they spin. PBR is like giving the objects a "skin" that reacts to light. It calculates how shiny the metal is, how rough the wood feels, and how light bounces off the glass.
• The Result: When you shine a virtual light on your generated scene, the metal cup reflects the light realistically, and the wooden table casts a soft shadow, just like in the real world.

Why Does This Matter?

ZeroScene is like a universal translator between the 2D world (photos) and the 3D world (games, VR, robots).

• For Gamers: You could take a photo of your living room and instantly turn it into a level for a video game.
• For Robots: Robots need to practice in virtual worlds before entering real ones. ZeroScene can turn a photo of a messy kitchen into a perfect training simulation for a robot chef.
• For Designers: You can take a sketch of a furniture arrangement and instantly see it as a realistic 3D room with editable textures.

In short, ZeroScene takes a single snapshot, understands the hidden 3D world inside it, fixes the missing pieces, and lets you remix the textures, all without needing a team of 3D artists or a massive database of training data. It turns a flat picture into a living, breathing 3D world.

Technical Summary

1. Problem Statement

Current 3D content generation methods face significant challenges when transitioning from single-object generation to complex scene reconstruction from a single RGB image. Key issues include:

• Scene Coherence vs. Asset Quality: Existing methods struggle to simultaneously ensure high-quality individual assets and the global coherence of the scene. They often fail to model spatial relationships, leading to physically implausible layouts (e.g., floating objects, mutual penetration, or incorrect support structures).
• Occlusion and Detail Loss: Single-view inputs suffer from severe occlusion, causing generated assets to lack structural integrity or exhibit multi-view inconsistencies.
• Texture Editing Limitations: While texture editing based on 3D geometry exists, current methods often suffer from the "multi-face Janus problem" (inconsistent views), blurriness, seam artifacts, and a lack of physical realism (PBR materials).
• Background Neglect: Most approaches focus solely on foreground objects, ignoring the background environment, which is crucial for scene context, lighting, and collision detection.

2. Methodology

ZeroScene is a zero-shot framework that leverages pre-trained large vision models to reconstruct a complete 3D scene (foreground + background) and supports controllable texture editing. The pipeline is divided into three main stages:

A. Foreground Object Generation and Composition

1. Instance Segmentation & Inpainting: The system decouples the input image into foreground and background. It detects foreground instances and uses Vision-Language Models (VLMs) to identify missing occluded regions. These regions are inpainted to create structurally complete 2D images for each object.
2. 3D Asset Generation: The inpainted 2D instances are converted into 3D models (meshes with PBR materials) using open-source image-to-3D models (e.g., Hunyuan3D 2.5).
3. Point Cloud Extraction: Using DUSt3R, the system estimates depth and camera parameters to generate a dense point cloud of the entire scene. It then segments this into individual instance point clouds based on the foreground masks.
4. Layout Optimization: To align the generated 3D assets with the input image's spatial layout, ZeroScene optimizes the pose (translation, rotation, scale) of each object. It minimizes a joint loss function combining:
   • 3D Chamfer Distance ($L_{3D}$): Aligns the generated object point cloud with the target scene point cloud.
   • 2D Projection Chamfer Distance ($L_{2D}$): Projects both clouds onto the 2D image plane to ensure alignment with the original image perspective, resolving depth/scale ambiguity.

B. Background Handling

Recognizing that backgrounds provide physical support and context:

1. Background Reconstruction: The system removes foreground instances (and their shadows) from the input image using VLMs to obtain a clean background image.
2. Plane Fitting: It re-estimates the background point cloud and fits planar surfaces (walls, floors) using RANSAC and constrained least squares.
3. Integration: The background mesh is optimized using the same layout loss as foreground objects, ensuring a unified, physically plausible scene assembly.

C. Controllable Texture Editing

ZeroScene allows users to edit object textures via text prompts while maintaining multi-view consistency:

1. Geometry-Aware Conditioning: The system renders normal maps, position maps, and edge maps from the mesh. These are concatenated and fed into a ControlNet alongside the text prompt to guide a diffusion model, ensuring "texture follows geometry."
2. Mask-Guided Progressive Generation: To solve multi-view inconsistency, the system generates images view-by-view. For a new view, it projects known textures from previous views to create an incomplete image. A binary mask identifies unknown (occluded) regions. The diffusion model performs inpainting on these masked regions, ensuring seamless continuity with existing views.
3. Back-Projection & PBR: The generated multi-view RGB images are preprocessed (delighting, super-resolution) and back-projected onto the UV space using a confidence-weighted fusion strategy (excluding grazing angles and edges). Finally, Physically Based Rendering (PBR) material maps (albedo, roughness, metallic) are estimated to enhance realism.

3. Key Contributions

• Holistic Scene Reconstruction: A novel framework that generates both foreground assets and background environments from a single image, ensuring spatial coherence and physical plausibility through joint 3D/2D optimization.
• Robust Layout Recovery: The introduction of a dual-space optimization strategy (3D point cloud alignment + 2D projection constraints) effectively recovers complex spatial relationships and resolves scale/depth ambiguities.
• Consistent Texture Editing: A mask-guided progressive generation strategy combined with geometric constraints enables high-fidelity, multi-view consistent texture editing that adheres to user text prompts.
• PBR Integration: The system estimates full PBR material maps, significantly improving rendering realism compared to methods that only generate albedo.
• Zero-Shot & Decoupled Design: The framework is modular, allowing for the flexible replacement of foundational models (segmentation, inpainting, 3D generation) without retraining.

4. Experimental Results

The authors evaluated ZeroScene on a dataset of multi-object scenes (real photos and 3D-FRONT synthetic data) and compared it against state-of-the-art methods (Hunyuan3D 2.5, MIDI, SceneGen, TEXTure, MVPaint, etc.).

• Quantitative Performance:
  • Geometry: ZeroScene achieved the lowest Chamfer Distance (CD) and highest F-Score at both object and scene levels (e.g., Scene CD: 0.0137 vs. 0.0384 for MIDI), indicating superior geometric accuracy.
  • Visual Quality: It outperformed baselines in CLIP and DINOv2 similarity scores, demonstrating better alignment with input images and text prompts.
  • Texture Generation: In texture editing tasks, ZeroScene achieved the lowest FID (42.19) and KID scores and the highest CLIP similarity (0.826), indicating textures closer to real-world distributions and user intent.
• Qualitative Improvements: Visual comparisons showed ZeroScene produces structurally complete objects without redundant artifacts, handles occlusions better, and generates seamless, high-resolution textures without the "Janus" problem or seam artifacts common in other methods.
• Ablation Studies: Proved that removing image inpainting, 3D/2D joint loss, or the mask-guided generation strategy significantly degrades geometric accuracy and texture consistency.

5. Significance and Applications

ZeroScene represents a significant leap forward in 3D content creation, bridging the gap between 2D images and immersive 3D worlds.

• Digital Twins: Enables rapid reconstruction of real-world environments from single photos for digital twin applications.
• Game Development: Provides a pipeline for generating diverse, high-quality 3D assets and scenes, reducing reliance on manual modeling.
• Robotics (Real-to-Sim): Facilitates the creation of highly realistic, trainable simulation environments for embodied AI, addressing the "sim-to-real" gap.
• Immersive Media: Supports the creation of VR/AR content with physically accurate lighting and materials.

By effectively handling occlusion, spatial relationships, and texture consistency in a zero-shot manner, ZeroScene offers a versatile tool for transforming 2D visual data into complex, editable 3D environments.