DreamAnywhere: Object-Centric Panoramic 3D Scene Generation

DreamAnywhere is a modular system that generates immersive, editable 360° panoramic 3D scenes from text by synthesizing a panoramic image, decomposing it into background and objects, and reconstructing detailed 3D representations, thereby enabling rapid prototyping and high-fidelity scene exploration for applications like low-budget film production.

The Gist

Imagine you want to build a magical, fully explorable 3D world just by typing a sentence like, "A mystical beach with giant crabs wearing hats."

In the past, AI could only paint a flat picture of this scene. If you tried to walk around the giant crabs in your imagination, the picture would warp, glitch, or fall apart. Other methods could build 3D rooms, but they often felt like cardboard cutouts that looked great from the front but collapsed if you looked at them from the side.

DreamAnywhere is a new system that solves this by acting like a master architect and a team of specialized construction workers working together. Here is how it works, broken down into simple steps:

1. The Blueprint: The 360° Panorama

Instead of trying to build the whole world at once, the system first draws a 360-degree panoramic photo (like a giant, seamless wallpaper that wraps around you).

• The Magic Trick: Usually, AI struggles to make these wide photos look consistent. DreamAnywhere uses a special "style guide" (a perspective image) to teach the AI exactly how the light, colors, and mood should look, ensuring the whole world feels like it belongs together.

2. The Separation: Cutting Out the Stars

Once the panoramic wallpaper is done, the system acts like a skilled editor with a pair of scissors.

• It identifies the "stars" of the show (the giant crabs, the trees, the rocks) and carefully cuts them out.
• This leaves behind a clean, empty background (the beach, the sky, the sand).
• Why do this? It's much easier to build a perfect 3D crab if you focus on just the crab, rather than trying to build the crab and the beach simultaneously.

3. The "Rescue Mission": Fixing the Crabs

Sometimes, the cut-out crab looks a bit weird—maybe it's blurry, or the hat is missing a piece because of the cut.

• DreamAnywhere doesn't just use the blurry cut-out. It sends the crab to a "Resynthesis Station."
• It asks a smart AI: "Based on the text description and the shape we have, what does a perfect, high-definition version of this crab look like?"
• It generates a brand new, crystal-clear image of the crab from all angles, then turns that into a 3D model. This ensures the crab looks solid and real, not like a flat sticker.

4. Filling the Holes: The 3D Inpainting

Now, imagine you have a 360° photo of a beach, but you've cut out the crabs. You have big holes in the sand where they used to be.

• The system uses a technique called "3D Inpainting." Think of this like a magical painter that doesn't just paint over the hole on the flat photo, but actually builds new sand and rocks in 3D space to fill the gap.
• It ensures that if you walk around the spot where the crab used to be, the sand looks real and consistent, not like a flat painting.

5. The Assembly: Putting It All Together

Finally, the system takes the high-quality 3D crabs and places them back onto the 3D beach.

• It uses "gravity" logic to make sure the crabs sit on the sand, not floating in the air.
• It even adds shadows so the crabs look like they are actually touching the ground.

The Result: A World You Can Walk Through

The final output is a 3D Gaussian Splat (a fancy term for a cloud of millions of tiny, colored dots that look like a photo but act like a 3D object).

Why is this a big deal?

• No More "Flat" Worlds: You can walk 100 meters away from the giant crab, look back, and it still looks 3D and correct. The world doesn't warp or glitch.
• Easy Editing: Because the system built the world in parts (background + individual objects), you can easily swap the "crab wearing a hat" with a "crab wearing a crown" without rebuilding the whole beach.
• Speed: It does all this in about 15 minutes, which used to take human artists days or weeks.

In short: DreamAnywhere is like a LEGO set for the AI world. Instead of trying to mold a giant statue out of clay (which cracks easily), it builds the world by creating a perfect base, crafting perfect individual pieces, and snapping them together so you can explore the result from any angle.

Technical Summary

1. Problem Statement

Current text-to-3D scene generation methods face significant limitations:

• Lack of Global Coherence: Existing approaches often generate scenes that are only coherent within small camera ranges. Large camera offsets lead to structural collapse, hallucinations, and semantic drift.
• Limited Interactivity: Most methods produce "monolithic" representations (e.g., meshes or single 3D volumes) that are difficult to edit, isolate, or manipulate at the object level.
• Domain Constraints: Many models are fine-tuned specifically for indoor or outdoor settings and struggle with out-of-distribution styles or complex compositions.
• Viewpoint Limitations: While video diffusion models offer temporal coherence, they struggle to maintain 3D consistency over long camera trajectories required for immersive exploration.

2. Methodology

DreamAnywhere proposes a modular, object-centric pipeline that uses a 360° panoramic image as an intermediate representation to generate, decompose, and reconstruct 3D scenes. The pipeline consists of three main stages:

Stage 1: Panoramic Image Generation & Decomposition

• 360° Panorama Diffusion: The system generates a 360° panoramic image ($I_F$) from a text prompt. To improve style fidelity and generalization, it employs a joint fine-tuning strategy. A LoRA (Low-Rank Adaptation) is trained on panoramic data, while an IP-Adapter mechanism (using cross-attention) conditions the generation on a perspective image derived from the same prompt. This bridges the gap between perspective and panoramic distributions.
• Instance Segmentation: Using Grounded-SAM (enhanced with GPT-4V for better class recognition), the system segments foreground objects from the background.
• 2D Inpainting (Background Extraction): Foreground objects are erased from the panorama. A hybrid inpainting strategy is used:
  1. Coarse Inpainting: Uses LaMA to create a coherent object-free initialization.
  2. Refined Inpainting: A custom LoRA fine-tuned on the Structured3D dataset fills in details, followed by Poisson blending to ensure wrap-around consistency.
• Depth Alignment: Depth maps for the full panorama ($I_F$) and the background ($I_B$) are estimated using different models (EGformer for indoor, 360MonoDepth for outdoor). A per-pixel shift and global scale alignment are performed to ensure objects and the background share a unified coordinate system.

Stage 2: High-Fidelity Object Reconstruction

Instead of directly lifting low-quality segmented crops to 3D, DreamAnywhere introduces a resynthesis step to ensure geometric and stylistic consistency:

• Reference Image Generation: A high-resolution reference image for each object is synthesized using multimodal cues:
  • Text: Descriptions generated by a Vision-Language Model (VLM) from the crop.
  • Geometry: Partial depth maps derived from the segmentation mask and scene depth.
  • Style: The original style image from the panorama generation.
• 3D Object Generation:
  1. Multi-view Synthesis: Zero123++ generates six multi-view images based on the reference.
  2. NeRF to 3DGS: InstantMesh converts these views into a NeRF representation (chosen for robustness during learning), which is then converted to 3D Gaussian Splatting (3DGS) for efficient rendering.
  3. Refinement: A VLM ranks multiple generated candidates to select the best 3D object.
• Pose Estimation: The system calculates the absolute pose of the original crop relative to the camera and the relative transformation between the original crop and the high-quality 3D object (using MAST3R). This ensures the reconstructed object is placed correctly in the scene.

Stage 3: 3D Background Generation & Composition

• 3D Background Initialization: The background panorama ($I_B$) is unprojected into a 3DGS point cloud.
• Hybrid 3D Inpainting: To handle disocclusions (areas hidden by objects in the original view but visible from new angles):
  1. Incremental Inpainting: The system identifies missing regions in novel views and uses a diffusion model fine-tuned for 3DGS depth inpainting to generate new Gaussians.
  2. Multi-view Fine-tuning: Score Distillation Sampling (SDS) is applied to ensure consistency across multiple viewpoints, removing artifacts and aligning new Gaussians with the existing scene.
• Final Composition: Objects are placed onto the background point cloud. Post-processing includes splat-aware shadow mapping to restore contact shadows lost during object removal and automatic snapping of objects to support planes (floors/walls) to ensure physical plausibility.

3. Key Contributions

1. Unified Modular System: A framework that separates background and object generation, enabling high 3D coherence, visual fidelity, and intuitive object-level editing.
2. Perspective-Conditioned Panorama Generation: A novel method that jointly fine-tunes a 360° diffusion model with a perspective conditioning mechanism (IP-Adapter), significantly improving out-of-domain style transfer.
3. Robust Object Reconstruction: A pipeline that synthesizes high-quality reference images from multimodal cues (text, geometry, style) before 3D reconstruction, overcoming issues with low-resolution or distorted segmentation masks.
4. Hybrid Inpainting Strategy: A combination of 2D inpainting (for global background coherence) and 3D inpainting (for local disocclusions), ensuring visual consistency under large camera offsets.

4. Results

• Quantitative Evaluation: On 17 text prompts, DreamAnywhere outperforms state-of-the-art methods (Text2Room, DreamScene360, LayerPano3D) in perceptual image quality (CLIP-IQA+, Q-Align, A-Align) and aesthetic metrics.
• Novel View Synthesis: The method demonstrates superior coherence when viewed from large camera offsets, avoiding the "flat" or "stitched" artifacts common in competing methods.
• User Study: A study with 28 participants showed a statistically significant preference ($p < .001$) for DreamAnywhere over existing methods across all metrics: Coherence, Compatibility, Immersiveness, and Overall Preference. Users specifically praised its geometric structure and navigability.
• Efficiency: Scenes are generated in approximately 15 minutes on a single H100 GPU.

5. Significance

DreamAnywhere addresses a critical gap in generative 3D content creation by enabling immersive, navigable, and editable 3D scenes from simple text prompts.

• Applications: It is particularly valuable for low-budget movie production, rapid prototyping, visual mock-ups, and VR/AR applications where traditional 3D modeling is too time-consuming.
• Technical Impact: By decoupling object generation from background generation and utilizing 3D Gaussian Splatting, the system achieves a level of structural coherence and editability that monolithic approaches cannot match.
• Future Potential: The modular nature of the pipeline allows for easy integration of future improvements in diffusion models, depth estimation, or 3D reconstruction techniques, making it a robust foundation for future world-to-world transfer models.