🤖 AI

Native3D: End-to-End 3D Scene Generation via Unified Mesh-Texture Modeling and Semantic Alignment

Native3D introduces the first end-to-end 3D scene generation framework that bypasses 2D intermediates by utilizing a unified mesh-texture joint representation and a novel 3D Representation Alignment Loss to achieve superior geometric and textural fidelity.

Original authors: Yibo Liu, Ziwei Zhang, Haozhou Pang, Menghao Li, Lanshan He, Gan Qi

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yibo Liu, Ziwei Zhang, Haozhou Pang, Menghao Li, Lanshan He, Gan Qi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to redesign your living room. In the past, if you asked a computer to "move the sofa" or "add a cartoon-style wall," it had to play a tricky game of translation. It would first try to understand your request in 2D (like a flat picture), then try to stretch and warp that flat image into a 3D room. This is like trying to build a real house by first drawing it on a piece of paper and then trying to pop the paper up into a 3D shape. Often, the walls would warp, the furniture would look blurry, or the sofa might end up floating in mid-air because the computer lost track of the 3D rules.

Native3D is a new tool that skips the "flat paper" step entirely. Instead of translating 2D pictures into 3D, it builds the 3D room from the ground up, understanding the shape and the texture at the same time.

Here is a simple breakdown of how it works and why it's different:

1. The "Unified Blueprint" (Mesh-Texture Joint Modeling)

Think of traditional 3D modeling like building a house where you first build the wooden frame (the geometry/mesh) and then, separately, you hire a painter to put wallpaper on it (the texture). If the frame shifts even a little, the wallpaper rips or looks wrong.

Native3D treats the frame and the wallpaper as one single, inseparable unit. It uses a special "brain" (a Transformer) that looks at the shape of the room and the colors/patterns on the walls simultaneously. This ensures that when you move a wall, the wallpaper stretches perfectly with it, and when you add a new chair, it fits into the space with the right texture from the very first moment.

2. The "Direct Translator" (No 2D Detour)

Most other AI tools try to use pre-trained artists who are experts at drawing 2D pictures. To make a 3D room, these tools force the 3D data to look like a 2D picture, let the artist draw on it, and then try to turn it back into 3D. This is like trying to explain a 3D sculpture to a 2D painter and hoping they get the depth right. It often leads to "domain gap" issues—where the 3D structure gets distorted or the details get muddy.

Native3D speaks "3D" natively. It doesn't force the data into a 2D shape to use the artist's help. Instead, it uses a powerful engine (called a Diffusion Transformer) that learns to generate 3D shapes directly. It's like hiring an architect who understands 3D space intuitively, rather than a painter who only knows how to paint on a flat canvas.

3. The "Quality Control Inspector" (3D REPA Loss)

Even with a great architect, sometimes the details can get fuzzy. To fix this, the authors added a special "Quality Control Inspector" called the 3D REPA Loss.

Imagine you are building a Lego set. You have the final picture (the goal) and the pieces you are building. The Inspector looks at your finished Lego house and compares it to the goal picture, but it checks two things at once:

The Big Picture: Does the whole room look right?
The Tiny Details: Is the texture on the specific chair sharp and correct?

If the computer tries to build a chair that looks like a blob, the Inspector says, "No, that doesn't match the texture of a real chair," and guides the AI to fix it. This ensures that the final room isn't just a vague shape, but has crisp, high-quality details.

What Can It Do?

The paper shows that this system can handle four main tasks based on simple text instructions:

Add: "Put a sofa in the corner." (The AI knows exactly where the corner is and what a sofa looks like in 3D).
Move: "Move the bed to the other side." (The AI shifts the bed without breaking the floor or the walls).
Remove: "Take the table away." (The AI removes the table and fills in the floor where it used to be so it looks natural).
Style: "Make the whole room look like a cartoon." (The AI changes the textures and colors of everything while keeping the room's structure intact).

The Bottom Line

The authors tested Native3D against other methods and found that it creates cleaner, more consistent 3D rooms. It doesn't suffer from the "glitchy" artifacts (like floating objects or warped walls) that happen when you try to force 2D tools to do 3D work.

In short, Native3D is like a master builder who understands the laws of physics and design natively, allowing you to simply say, "Make this room look like this," and getting a result that is structurally sound and visually sharp, without the computer having to guess how to turn a flat idea into a 3D reality.

Technical Summary: Native3D

Problem Statement

The automated generation and editing of high-quality 3D indoor scenes face significant barriers in traditional workflows, including the need for specialized expertise, expensive software, and lengthy production timelines. While deep generative models have advanced the field, existing approaches suffer from a fundamental domain gap between 2D and 3D representations.

Current methods generally fall into two paradigms, both of which have critical limitations:

Pure 2D Approaches: These treat scene editing as a 2D pixel-space problem. They lack explicit 3D geometric constraints, leading to cross-view inconsistencies, blurring artifacts, and visible seams when viewed from different angles. They cannot inherently reason about 3D structure, depth, or spatial topology.
Hybrid 2D-3D Approaches: These use 2D diffusion models for editing and then reconstruct 3D geometry (e.g., via NeRF or 3D Gaussian Splatting). They struggle with consistency propagation between 2D editing and 3D reconstruction, often resulting in fine-grained misalignment, slow convergence, and geometric distortion due to the inherent mismatch between 2D priors and 3D requirements.

The core challenge is achieving robust multi-view consistency and geometric fidelity without compromising computational practicality or editing versatility, a goal currently hindered by the reliance on 2D intermediate representations.

Methodology: Native3D

The authors propose Native3D, the first end-to-end 3D-native generation framework designed to completely bypass 2D intermediate representations. The method operates directly on 3D mesh geometry and texture information.

1. Unified Mesh-Texture Joint Representation

To address the disconnect between geometry and appearance, Native3D introduces a unified representation that models both simultaneously:

Input: A 3D scene $S$ is represented as the union of mesh grids ( $M_i$ ) and texture features ( $T_i$ ) for all $N$ objects in the scene.
Encoder: A hierarchical feature encoder based on a Transformer architecture is used. It employs a local encoder (adapted from pre-trained HunYuan3dShapeVAE) to process individual object meshes and textures, followed by an adapter network ($Adpt$) that uses self-attention to model inter-object relationships and aggregate global scene features.
Output: This joint representation captures global semantic information, local geometric details, and precise spatial positioning.

2. 3D Diffusion Process

The generation core utilizes a Diffusion Transformer (DiT):

Architecture: The system adopts the HunYuanDiT as the backbone and uses Flow Matching as the sampler.
Process: The diffusion process occurs directly in the high-level 3D feature space. This eliminates information loss caused by 2D-3D cross-domain conversion.
Conditioning: Natural language descriptions serve as conditional control signals, allowing users to specify complex tasks (e.g., "Add a sofa," "Remove the bed," "Apply cartoon-style") via text embeddings generated by an umT5 encoder.
Decoding: A specifically designed decoder reconstructs the output of the DiT module into standard 3D mesh and texture representations.

3. 3D REPA Loss (Representation Alignment)

To enhance local detail fidelity and ensure semantic coherence, the authors propose the 3D Representation Alignment Loss (3D REPA Loss):

Mechanism: This loss employs an improved contrastive learning mechanism. It uses a frozen Direct3D Encoder to extract multi-level semantic representations from both the generated results and ground truth samples.
- For generated models, it extracts global scene features.
- For ground truth, it extracts both scene-level global features and object-level local features.
Alignment: The loss aligns these multi-scale representations in the latent space using a modified InfoNCE loss. This encourages the model to maintain the discriminative nature of individual object features while ensuring semantic consistency at the scene level.
Regularization: To mitigate training instability caused by distribution mismatches, a latent space regularization term ( $\ell_2$ -norm constraint) is added to the overall optimization objective.

Key Contributions

The paper claims three primary contributions:

Native3D Framework: The first end-to-end 3D generation framework that completely bypasses 2D intermediate representations, eliminating domain gap issues at the architectural level.
Unified Feature Representation: A novel joint modeling mechanism for mesh geometry and texture appearance, enabling the simultaneous capture of global layouts and local details within a single 3D-native pipeline.
3D REPA Loss: A novel loss function that aligns multi-level semantic representations to significantly improve geometric and textural fidelity, guiding the model to accurately restore details while maintaining global consistency.

Experimental Results

The authors evaluated Native3D on a dataset of 9,001 room pairs (derived from 3D-FRONT) with editing instructions covering four categories: Add, Remove, Move, and Style.

Quantitative Performance

Overall Quality: Native3D achieved the best CLIP Score (CS) (32.1) among compared methods (Text2tex, SceneTex, RoomTex, RoomPainter), indicating superior semantic understanding and alignment with text prompts. It also achieved the highest average ranking (1.6) across metrics.
Editing Capabilities: In the ImgEdit-Bench evaluation (measuring Instruction Adherence, Editing Quality, and Detail Preservation), Native3D achieved state-of-the-art performance in structural editing tasks:
- Add: 4.01 (vs. 2.87 for the next best)
- Remove: 4.14 (vs. 2.99)
- Move: 2.61 (vs. 2.24)
- Style: 3.63 (competitive, though RoomTex scored higher at 4.14).
Ablation Studies: Removing the 3D feature alignment or the 3D REPA Loss resulted in significant performance drops and visual artifacts (e.g., object penetration, failure to follow instructions), validating the necessity of these components.

Qualitative Observations

Visual results demonstrated the model's ability to:

Cleanly eliminate target objects while reconstructing underlying surfaces.
Relocate objects with appropriate spatial relationships and lighting consistency.
Apply style transfers that alter textures and materials while preserving the fundamental layout and geometry.

Significance and Claims

The paper positions Native3D as a novel solution that bridges the domain gap between 2D priors and 3D generation. By operating natively in 3D, the method provides:

Robust Multi-View Consistency: Eliminating the cross-view inconsistencies and artifacts common in 2D-based methods.
Geometric and Textural Fidelity: Preserving structural integrity and fine-grained details through the unified mesh-texture modeling and 3D REPA Loss.
Versatile Editing: Enabling intuitive, natural language-based control for complex scene modifications (addition, removal, rearrangement, and stylization).

The authors conclude that while the method excels in structural and semantic accuracy, there remains room for improvement in photorealistic rendering and domain generalization, particularly regarding illumination and camera parameters. Nevertheless, the framework offers a promising direction for applications in interior design, virtual reality content creation, and professional 3D design workflows.