Extend3D: Town-Scale 3D Generation

1. Problem Statement

The paper addresses the challenge of generating large-scale, detailed 3D scenes from a single 2D image. Current state-of-the-art 3D generative models face two primary limitations:

• Object-Centric Bias: Most models are trained on datasets containing single objects rather than complex scenes with multiple interacting objects and backgrounds. Consequently, they struggle to generate coherent large-scale environments.
• Fixed Latent Size: Existing latent generative models use a fixed-size latent space. As the desired scene scale increases, the fixed resolution leads to blurry, low-detail outputs. Expanding the latent space to accommodate larger scenes is non-trivial because standard models cannot handle variable dimensions without retraining.

Previous "training-free" approaches (e.g., outpainting) often result in visible seams, inconsistent geometry, and repeated objects because they generate sub-scenes sequentially without sufficient spatial coupling.

2. Methodology: Extend3D

Extend3D is a training-free pipeline that adapts a pre-trained object-centric 3D generative model (specifically Trellis) to generate town-scale scenes. The method consists of three core components:

A. Extended Latent Space & Overlapping Patch-wise Flow

To overcome the fixed-size limitation, the authors extend the latent space in the $x$ and $y$ dimensions.

• Patch Division: The extended latent space is divided into overlapping patches using a sliding window.
• Coupled Denoising: Instead of generating patches sequentially, the model generates them simultaneously. Each patch is conditioned on a corresponding crop of the input image.
• Vector Field Merging: The vector fields (denoising directions) for each patch are computed independently and then merged. In overlapping regions, the vector fields are averaged. This allows neighboring patches to influence each other, correcting local errors and ensuring global consistency without visible seams.

B. Initialization with Priors & Under-Noising

To address the misalignment between 2D images and 3D latents, and the inability of object-centric models to handle occlusions:

• Point Cloud Prior: The scene is initialized using a point cloud extracted from a monocular depth estimator (MoGe-2). This provides a rough 3D structure aligned with the input image.
• Iterative SDEdit: The method uses SDEdit (Stochastic Differential Equation editing) to refine the initialized structure.
• Under-Noising: A novel technique where the latent is "under-noised" ($t_{noise} < t_{start}$). By adding less noise than the standard SDEdit process, the model treats missing or occluded regions (holes in the point cloud) as "additional noise" that needs to be filled during the aggressive denoising phase. This effectively completes occluded areas rather than just refining existing ones.

C. 3D-Aware Optimization

To prevent the generated sub-scenes from reverting to "object-like" dynamics (e.g., floating floors, vanishing objects) and to ensure texture fidelity:

• Latent Optimization: At every denoising time step, the extended latent vector is optimized using gradient descent.
• Sparse Structure Loss ($\mathcal{L}_{SS}$): Ensures that voxels corresponding to the initial point cloud do not disappear during denoising.
• Structured Latent Loss ($\mathcal{L}_{SLAT}$): Uses a differentiable renderer (Gaussian Splatting) to render the current 3D state and compares it to the input image using LPIPS and SSIM. This optimizes the texture and geometry to match the image condition and aligns patch boundaries.

3. Key Contributions

1. Scalable Latent Extension: Successfully extends the latent space of pre-trained object-centric models to generate town-scale scenes without retraining the base model.
2. Overlapping Patch-wise Flow: Introduces a coupled denoising mechanism that captures local information and mitigates errors (seams, repetitions) through overlapping patch interactions.
3. Under-Noising & Iterative SDEdit: Proposes a specific noise schedule for 3D completion that treats occluded regions as noise to be filled, enabling robust structure completion from sparse priors.
4. 3D-Aware Optimization: Develops specific loss functions to align the denoising trajectory with scene dynamics and image conditions, ensuring geometric consistency and texture fidelity.

4. Experimental Results

The authors evaluated Extend3D against state-of-the-art methods including Trellis, Hunyuan3D, EvoScene, and SynCity.

• Human Preference: In a study with 10 annotators, Extend3D outperformed all baselines in four categories: Geometry, Faithfulness (to the image), Appearance, and Completeness.
• Quantitative Metrics:
  • Image Fidelity: Achieved the best scores in LPIPS (0.240), SSIM (0.611), and PSNR (20.4) when rendering the 3D scene back to the camera view.
  • Geometric Accuracy: Outperformed baselines in Chamfer Distance (CD) and F-score on the UrbanScene3D dataset.
  • Consistency: Showed superior patch-wise consistency (lower Intra-LPIPS) compared to sequential outpainting methods like SynCity.
• Qualitative Results: The method successfully generated complex scenes (e.g., Vatican City, medieval markets) with fine details, correct object placement, and no visible seams, whereas baselines often produced distorted geometry, missing floors, or repeated objects.

5. Significance

Extend3D represents a significant leap in generative 3D content creation. By demonstrating that pre-trained object-centric models can be effectively repurposed for large-scale scene generation through latent extension and optimization, it bypasses the need for expensive, large-scale 3D scene datasets. The "training-free" nature of the pipeline makes it highly accessible and adaptable, offering a practical solution for industries requiring rapid 3D asset generation (gaming, film, simulation) from single images. The introduction of under-noising and 3D-aware optimization provides new theoretical insights into how diffusion/flow models can be guided to complete complex 3D structures.