Map2World: Segment Map Conditioned Text to 3D World Generation

1. Problem Statement

While recent advances in 3D asset generation (e.g., TRELLIS, CLAY) have achieved high-quality results for individual objects, scaling these capabilities to generate entire 3D worlds remains a significant challenge. Existing approaches face three primary bottlenecks:

• Data Scarcity: High-quality, world-scale datasets are rare, forcing models to either train from scratch on limited data (resulting in poor generalization) or rely on pre-trained asset generators that lack world-level context.
• Inconsistency and Disconnectedness: Methods that stitch together generated assets (e.g., grid-based tiling like SynCity) often suffer from scale inconsistencies, lack of semantic continuity between adjacent tiles, and visible boundary artifacts.
• Rigid Layouts: Existing world generators often constrain scenes to fixed grid layouts, making it impossible to generate environments with irregular district boundaries or arbitrary shapes required for real-world applications like autonomous driving simulation.

2. Methodology: Map2World

Map2World is a novel framework that leverages the strong priors of the pre-trained 3D asset generator TRELLIS to synthesize large-scale, coherent 3D worlds conditioned on user-defined segment maps and text prompts. The pipeline consists of three core components:

A. Latent Fusion for Spatial Expansion (Section 4.1)

To overcome the resolution limits of the base model (which generates 64x64x64 cubes), Map2World employs a multi-diffusion strategy within the structured latent space (SLAT).

• Overlapping Windows: The 3D space is divided into overlapping cubic windows. The model aggregates velocity field predictions from all windows covering a specific position using a weighted Gaussian kernel. This ensures seamless transitions and global consistency without explicit boundary blending.
• Segment-Map Guidance: The diffusion process is conditioned on a binary segment map ($M_k$) and corresponding text prompts ($y_k$). The velocity at any point is a weighted sum of velocities derived from different semantic regions, allowing for arbitrary shapes and non-grid layouts.
• Scale-Aware Initialization: To ensure global scale consistency, the authors optimize the initial noisy latent. They observe that sparse structures with similar scales cluster in the latent space. By optimizing the initial noise in the spectral domain (using 3D FFT), they can steer the generation toward a desired scale efficiently and stably.

B. Detail Enhancer Network (Section 4.2)

The initial latent fusion generates a coarse global structure but lacks fine-grained details due to the limited capacity of the latent space.

• Latent Super-Resolution: Instead of training a new generator from scratch, the authors design a Detail Enhancer that learns the mapping from a coarse latent (representing a large cube) to fine-grained latents (representing smaller sub-cubes).
• Architecture: The enhancer utilizes a Flow Transformer (from TRELLIS) augmented with a lightweight MLP layer.
  • Inputs: It takes the coarse latent of the target region ($s^{O|j}$) and the latents of adjacent cubes ($s^{Adj(j)}$) as conditions.
  • Mechanism: The MLP mixes the noise and condition latents before feeding them into the Transformer. This allows the model to inject fine details while preserving the global geometry and ensuring seamless connections between adjacent regions.
• Training Strategy: The model is fine-tuned using a small number of parameters (only the MLP weights) on a dataset constructed by splitting existing 3D scenes into large and small cubes. This preserves the generalization capabilities of the pre-trained TRELLIS model.

C. Decoder Fine-Tuning (Section 4.3)

Since the original TRELLIS decoder was trained on complete objects, it struggles with partial scene cubes. The authors fine-tune the SLAT Decoder ($D_L$) using the small cubes from the detail enhancer training set to ensure high-quality reconstruction of the enhanced latents.

3. Key Contributions

1. Flexible Segment Map Conditioning: Unlike previous grid-based methods, Map2World can generate 3D worlds from arbitrary-shaped segment maps, enabling complex, irregular layouts (e.g., organic city districts) while maintaining semantic alignment with text prompts.
2. Global Consistency via Latent Fusion: By applying multi-window diffusion in the structured latent space, the method ensures scale consistency and seamless connections across the entire world, avoiding the "patchwork" look of tiled approaches.
3. Detail Enhancement without Data Scarcity: The proposed Detail Enhancer leverages the priors of a powerful asset generator to add fine-grained details without requiring massive world-scale datasets, achieving high fidelity through latent-space super-resolution.
4. Robust Generalization: The framework achieves strong performance across diverse domains (e.g., forests, cities) even with limited training data for the scene generation task.

4. Experimental Results

The authors evaluated Map2World against state-of-the-art baselines, including SynCity and GaussianCube, using both qualitative and quantitative metrics.

• Qualitative Comparison:
  • Map2World successfully generates worlds with irregular shapes and large, connected structures (e.g., skyscrapers spanning multiple segments), whereas SynCity fails to generate coherent objects across tile boundaries and produces disconnected assets.
  • The generated worlds exhibit smooth transitions between different semantic regions (e.g., forest to city) without visible seams.
• Quantitative Metrics (World Quality - WQ):
  • A new metric, World Quality (WQ), was introduced, weighting World Completeness (45%), Coherence (25%), Realism (15%), and Sharpness (15%).
  • Results: Map2World achieved a WQ score of 7.76, significantly outperforming SynCity (7.25) and GaussianCube (5.08).
  • Specifically, Map2World excelled in World Completeness (7.8 vs. 6.8) and Coherence (7.9 vs. 7.6), demonstrating its ability to create structurally consistent, large-scale environments.
• Ablation Studies:
  • Spectral Parameterization: Crucial for stable scale control; without it, optimization diverged or required excessive steps.
  • Detail Enhancer Design: The proposed concatenation architecture with adjacent cube conditioning outperformed IP-Adapter and methods using Classifier-Free Guidance (CFG), which caused geometric distortions.

5. Significance

Map2World represents a paradigm shift in 3D world generation by moving away from rigid grid-based tiling toward flexible, segment-map-conditioned synthesis. Its ability to generate globally consistent, high-resolution 3D environments with arbitrary shapes makes it highly valuable for:

• Autonomous Driving Simulation: Generating diverse, realistic, and complex road networks and urban layouts.
• Immersive Content Creation: Allowing creators to design vast virtual worlds with specific semantic layouts using simple text and map inputs.
• Scalable Generation: Providing a pathway to generate world-scale content by leveraging the priors of asset-level models, effectively bypassing the need for massive, non-existent world-scale datasets.

In summary, Map2World bridges the gap between high-quality 3D asset generation and scalable world synthesis, offering a controllable, consistent, and detailed solution for next-generation 3D content creation.