WorldMesh: Generating Navigable Multi-Room 3D Scenes via Mesh-Conditioned Image Diffusion

WorldMesh introduces a geometry-first framework that generates scalable, navigable multi-room 3D scenes by first constructing a mesh scaffold to define structural consistency and then leveraging mesh-conditioned image diffusion to populate the environment with photorealistic objects and layouts.

The Gist

Imagine you want to build a massive, realistic virtual house with many rooms, but you only have a single sentence describing it, like "a cozy, sunlit Scandinavian apartment."

In the past, AI tried to do this by just "dreaming" up pictures from scratch. It was like asking an artist to paint a whole city by looking at a blank canvas and guessing what the buildings look like from every angle. The problem? The artist would get confused. The kitchen might look great from the front, but if you walked around the table, the table might suddenly vanish or turn into a chair. The walls might warp, and the rooms wouldn't connect properly.

WorldMesh is a new method that solves this by changing the order of operations. Instead of painting the picture first, it builds the skeleton first.

Here is how it works, using a simple analogy:

1. The Blueprint (The Mesh Scaffold)

Think of the AI first acting like an architect. Instead of trying to paint the whole house at once, it reads your text prompt and draws a strict 3D blueprint (called a "mesh scaffold").

• It decides exactly where the walls, floors, and doors go.
• It builds a digital wireframe of the house.
• Why this matters: This is the "skeleton" of the house. No matter how you walk through it, the walls stay in the same place. The rooms are connected correctly. This solves the problem of the house falling apart or looking different from different angles.

2. The Interior Designer (Object Placement)

Now that the skeleton is built, the AI needs to fill the rooms.

• It looks at the empty blueprint and asks, "What furniture goes here?"
• It uses a powerful image generator to create pictures of what the room could look like.
• It then takes those pictures, cuts out the furniture (like sofas and lamps), and physically places 3D versions of them into the blueprint.
• The trick: It makes sure the sofa sits on the floor and the lamp is under the ceiling, not floating in mid-air.

3. The Painter (Mesh-Conditioned Synthesis)

This is the magic step. Now the AI has a 3D skeleton with some furniture in it, but the walls are still blank.

• Instead of painting the whole room from scratch every time, the AI uses the skeleton as a guide.
• It "paints" the walls and textures the furniture, but it does so while looking at the 3D structure.
• The Analogy: Imagine you are painting a mural on a curved wall. If you just paint a flat picture, it will look warped when you walk past it. But if you paint directly onto the 3D wall (or use the wall's shape to guide your brush), the painting looks perfect from every angle.
• WorldMesh does this digitally. It generates images for different camera angles, but because they are all "anchored" to the same 3D skeleton, the lighting, colors, and textures stay consistent. If you walk from the living room to the bedroom, the style doesn't suddenly change.

4. The Quality Check (The "Reality Test")

Before finalizing the scene, the AI plays a game of "spot the difference."

• It generates an image and then asks, "Does this image match the blueprint?"
• If the AI accidentally draws a door where a wall should be, or if the depth looks wrong, it rejects that image and tries again.
• This ensures the final result isn't just a pretty picture, but a navigable 3D world you can actually walk through without hitting invisible walls or falling through the floor.

The Result

The final output is a 3D Gaussian Splat (a fancy term for a cloud of millions of tiny, colored dots that look like a photo but act like a 3D model).

In summary:

• Old Way: Try to imagine the whole world at once. Result: Confusing, inconsistent, and breaks when you move.
• WorldMesh Way: Build the frame first, fill in the furniture, then paint the walls while holding onto the frame. Result: A giant, consistent, multi-room world that looks real from every angle, generated just from a text description.

It's like the difference between a child scribbling a chaotic drawing of a house versus an architect building a solid model and then carefully decorating it. The result is a world you can actually explore.

Technical Summary

1. Problem Statement

Current text-to-3D and text-to-video generation methods struggle to create environment-scale, multi-room 3D scenes. While recent advances in image and video diffusion models (e.g., DreamFusion, WorldExplorer) can generate impressive single-room or object-level scenes, they face fundamental limitations when scaling to larger environments:

• Lack of Explicit Geometry: Purely 2D or video-based synthesis lacks a structural backbone, leading to geometric drift and inconsistencies.
• View Inconsistency: These methods fail to maintain coherent appearance across diverse viewpoints, particularly in close-up views or when transitioning between rooms.
• Scalability: Accumulated errors in iterative synthesis make it intractable to scale to arbitrarily large, complex multi-room layouts.

The core challenge is balancing global spatial consistency (the layout and structure across rooms) with local photorealistic appearance (textures, lighting, and object details).

2. Methodology: WorldMesh

WorldMesh proposes a geometry-first approach that decouples the complex task of 3D scene synthesis into two distinct stages: constructing a structural mesh scaffold and then conditioning image diffusion on this scaffold.

A. Mesh Scaffold Construction

The process begins by generating a geometric "scaffold" ($M$) that defines the scene's layout before any realistic textures are applied.

1. Layout Generation: An input text prompt ($T$) is processed by an LLM (Claude Opus) to generate a JSON floor plan ($L$) containing structural metadata (wall thickness, ceiling height, room definitions, doors/windows).
2. 3D Mesh Instantiation: The 2D floor plan is extruded into a 3D structural mesh ($M_{struct}$). This includes walls, floors, ceilings, and boolean operations to carve out openings.
3. Object Instantiation:
   • The structural mesh is rendered from a coverage camera to produce a depth map.
   • This depth map, combined with the text prompt, conditions an image synthesis model (Flux2-Klein) to generate a photorealistic image of the room with furniture.
   • SAM-3D-Objects is used to segment objects from this image and reconstruct them as 3D meshes with canonical coordinates and transforms.
   • These objects are placed into the scene, creating a geometrically populated scaffold ($M_{geo}$).
4. Initial Texturing: Projective texture accumulation is used to map initial wall textures onto the scaffold based on the generated images, providing a coarse but consistent color baseline.

B. Mesh-Anchored Photorealistic Appearance Synthesis

Once the scaffold is built, the system iteratively generates high-fidelity images for 3D reconstruction.

1. Viewpoint Strategy: A set of cameras is generated for each room. To ensure style consistency, the system uses a greedy nearest-neighbor strategy. It selects the next camera pose based on rotational similarity (quaternion dot product) to previously generated views, creating a gradual spiral that propagates style without drift.
2. Mesh-Conditioned Diffusion: An image synthesis model ($\Phi$) generates images conditioned on:
   • Depth Maps: Rendered from the mesh scaffold.
   • Alpha-Blended Textures: A composite of object textures and accumulated wall textures.
   • Style Reference: The previously generated image from the most rotationally similar camera pose.
3. Image Verification: To prevent the diffusion model from hallucinating geometry that contradicts the scaffold, a validation loop is employed. A depth estimation model predicts depth from the generated image, and Canny edge detection compares the structural edges of the predicted depth against the rendered mesh depth. Images failing this "edge recall" threshold are regenerated.
4. 3DGS Optimization: The final navigable scene is reconstructed using 3D Gaussian Splatting (3DGS). The optimization is regularized by the mesh scaffold's depth maps to prevent geometric drift, ensuring the final 3D scene adheres to the original structural layout while capturing the photorealistic details from the synthesized images.

3. Key Contributions

• First Scalable Multi-Room Synthesis: WorldMesh is the first approach to generate arbitrarily large, multi-room 3D scenes from text prompts by decoupling structure and appearance.
• Mesh-Conditioned Generation: It introduces a novel conditioning mechanism where a 3D mesh scaffold acts as a geometric anchor for image diffusion, ensuring multi-view consistency across both local (objects) and global (inter-room) scales.
• Linear Scalability: The approach scales linearly with the number of rooms, overcoming the exponential complexity and drift issues of purely pixel-space models.
• Robust 3D Consistency: By grounding generation in explicit geometry, the method solves the "flickering" and "morphing" artifacts common in existing text-to-3D methods when viewed from close proximity or during camera transitions.

4. Results and Evaluation

The authors evaluated WorldMesh against state-of-the-art baselines including WorldExplorer, FlexWorld, SpatialGen, WonderWorld, and DreamScene360.

• Qualitative Results: WorldMesh demonstrated superior performance in maintaining 3D consistency during close-up views and transitions between rooms. Baselines often suffered from object distortion, texture warping, or geometric drift in these scenarios.
• Quantitative Metrics:
  • Automatic Metrics: Achieved the highest scores in CLIP-IQA+ (technical quality) and CLIP Aesthetic (visual appeal).
  • Perceptual Study: In a study with 31 participants, WorldMesh significantly outperformed all baselines in 3D Object Consistency, 3D Structure Coherence, and Overall Quality.
  • Preference: Users preferred WorldMesh over the best baseline (SpatialGen) in 100% of pairwise comparisons, and over other methods in 93.5%–96.8% of cases.
• Ablation Studies: Experiments confirmed that the full pipeline (Depth + Textured Walls + Reconstructed Objects) is necessary. Removing any component (e.g., using only depth or only bounding boxes) led to significant inconsistencies in either object geometry or wall textures.

5. Significance

WorldMesh represents a significant step forward in environment-scale 3D world synthesis. By shifting from a purely generative (pixel-space) paradigm to a geometry-conditioned paradigm, it bridges the gap between the photorealism of diffusion models and the structural rigidity required for navigable 3D environments. This approach enables the creation of immersive, consistent, and complex virtual spaces for applications in virtual reality, video games, architecture visualization, and interior design, tasks that were previously too labor-intensive for manual modeling and too unstable for purely generative AI.

Limitations: The current method is limited to single-story layouts and relies on SAM-3D-Objects, which may produce incomplete geometry for occluded object regions. Future work aims to extend this to multi-level buildings and improve object reconstruction fidelity.