💻 computer science

MeshReGen: A Unified 3D Geometry Regeneration Framework

MeshReGen is a unified, self-supervised framework that regenerates 3D objects from initial shapes and images using a VecSet-based conditioning mechanism to achieve state-of-the-art performance in controllable 3D enhancement, reconstruction, and editing.

Original authors: Geon Yeong Park, Roman Shapovalov, Rakesh Ranjan, Jong Chul Ye, Andrea Vedaldi, Thu Nguyen-Phuoc

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Geon Yeong Park, Roman Shapovalov, Rakesh Ranjan, Jong Chul Ye, Andrea Vedaldi, Thu Nguyen-Phuoc

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a rough, blocky clay sculpture of a car. It has the right shape and size, but it's missing the curves, the shiny paint, the intricate grille, and the tiny details that make it look real. Now, imagine you also have a photograph of a real car that looks exactly like the one you want to build.

MeshReGen is like a magical, super-smart sculptor that takes your rough clay block and your reference photo, and instantly refines the clay into a perfect, high-definition replica. It doesn't just guess what the car should look like; it respects the rough shape you gave it while adding all the missing details to match your photo.

Here is how the paper explains this technology in simple terms:

The Big Problem: "One-Shot" vs. "Refining"

Most current AI tools for making 3D objects work like a "one-shot" artist. You give them a text description or a picture, and they try to create the whole object from scratch.

The Issue: If you want to fix a specific part of an object, or if you already have a rough draft (like a blurry scan or a simple block-out), these tools often ignore your draft and create something completely different. They lack "control."

The Solution: MeshReGen

The authors created MeshReGen, a system that doesn't start from scratch. Instead, it acts as a regenerator. It takes an existing, low-quality 3D shape (the "rough draft") and a 2D image (the "goal"), and it upgrades the draft into a high-quality, detailed version.

Think of it like Photoshop for 3D objects, but instead of just painting pixels, it rebuilds the actual geometry.

How It Works (The Magic Tricks)

1. The "Universal Translator" (VecSet)
The system uses a special language called VecSet to talk about 3D shapes.

The Analogy: Imagine you have a rough sketch of a house and a detailed blueprint. Usually, computers struggle to compare a sketch to a blueprint because they are written in different "languages." MeshReGen translates both the rough sketch and the final detailed shape into the same language (VecSet). This allows the AI to see exactly how the rough sketch needs to change to become the detailed version.

2. The "Smart Diffusion" Process
The system uses a technique called Diffusion, which is like slowly turning a blurry, noisy image into a clear one.

The Analogy: Imagine you have a muddy window (the rough 3D shape). You want to see the garden outside clearly (the detailed shape). MeshReGen doesn't just wipe the window; it uses the photo of the garden you're holding to guide the cleaning process. It knows exactly which parts of the mud to scrub away and which new details to paint in, all while keeping the window frame (the original shape) exactly where it was.

3. Learning Without a Teacher (Self-Supervised)
Usually, to teach an AI to do this, you need thousands of pairs of "Rough Shape + Perfect Shape" created by humans. That is expensive and hard to get.

The Analogy: Instead of hiring a teacher to show the AI every example, the researchers taught the AI to teach itself. They took a huge library of perfect 3D objects and artificially "ruined" them (made them blurry, chunky, or incomplete) to create the "rough drafts." The AI then learned how to fix these ruined versions back to their original glory. This way, it learned the rules of 3D geometry without needing any human labels.

What Can It Do?

The paper shows that this single "sculptor" can handle three different jobs, all using the same brain:

3D Enhancement: Taking a low-resolution, blocky 3D model (like one from an old video game) and making it look modern and crisp.
3D Reconstruction: Taking a messy, incomplete 3D scan of a real object (like a scan of a chair that has a hole in the back because the scanner couldn't see it) and filling in the missing parts to make a perfect, complete chair.
3D Editing: Taking a 3D object and changing a specific part of it based on a new photo. For example, if you have a 3D dog and you want to change its ears to be floppy, you can show the AI a picture of floppy ears, and it will reshape the 3D dog's ears to match, while keeping the rest of the dog exactly the same.

Why Is This Special?

The paper claims that previous tools were like having three different artists: one for fixing, one for scanning, and one for editing. MeshReGen is a unified framework, meaning it is one single tool that can do all three jobs equally well. It achieves this by understanding that whether you are fixing a scan, editing a model, or enhancing a block-out, the core task is the same: turning a "Low-Information" shape into a "High-Information" shape.

In short, MeshReGen is a versatile tool that lets you start with a rough idea or a messy scan and turn it into a professional-grade 3D object, guided by a simple picture, all while keeping your original structure intact.

Technical Summary: MeshReGen: A Unified 3D Geometry Regeneration Framework

Problem Statement

Current 3D generative models predominantly operate in a "one-shot" fashion, converting text or single images into 3D objects. While effective for initial creation, these models often lack controllability and struggle to integrate with complex, multi-stage 3D artist workflows. Real-world 3D creation frequently involves refining coarse geometry, reconstructing objects from incomplete scans (e.g., photogrammetry), or editing specific regions of an existing mesh. These tasks require mapping a "Low-information" 3D shape ( $y_L$ )—which may be coarse, noisy, distorted, or partial—to a "High-information" 3D shape ( $x_H$ ).

Existing approaches address these specific instances (enhancement, reconstruction, editing) using tailored, task-specific architectures. Furthermore, the scarcity of paired training datasets (pairs of low-quality and high-quality 3D shapes) limits the ability to train general-purpose regenerators. The paper identifies a gap in having a unified architecture and learning formulation that can handle diverse 3D regeneration tasks without requiring task-specific designs or expensive manual annotations.

Methodology

Core Concept: 3D Shape Regeneration

The authors define 3D regeneration as sampling a high-fidelity shape $x_H$ from the distribution $p(x_H | y_L, I)$ , where $y_L$ is an auxiliary low-information 3D shape (e.g., a coarse mesh, point cloud, or masked region) and $I$ is an optional 2D image guide.

Model Architecture

MeshReGen is built upon a pre-trained latent diffusion model operating in a compact 3D latent space using VecSet [78].

Latent Representation: Both the target high-quality shape ( $x_H$ ) and the conditioning low-quality shape ( $y_L$ ) are encoded into latent vectors ( $z_H$ and $z_L$ ) using the same VecSet encoder. VecSet represents 3D shapes as a set of $K$ latent vectors derived from a sampled point cloud.
Unified Conditioning Mechanism: Unlike prior works that use separate cross-attention pathways for conditioning (which can be inefficient for dense inputs), MeshReGen concatenates the conditioning tokens $z_L$ $z_{L}$ directly with the noisy target tokens $z_H^t$ $z_{H}^{t}$ .
- Token Pre-processing: To prevent the model from being "shocked" by the introduction of new control tokens, the conditioning tokens $z_L$ are passed through a Multi-Layer Perceptron (MLP) with zero-initialized weights (similar to ControlNet). Positional embeddings are also added to $z_L$ to help the model distinguish their spatial locations.
- Diffusion Process: A Diffusion Transformer (DiT) denoises the concatenated input $z_H^t \oplus \text{MLP}(z_L + \text{PE})$ conditioned on the image $I$ and time step $t$ .
Efficiency: A key advantage of VecSet is that the number of tokens $K$ can be adjusted based on shape complexity. Coarse inputs ( $y_L$ ) can be encoded with significantly fewer tokens ( $C \ll K$ ) than the target, improving computational efficiency due to the quadratic complexity of self-attention.

Data Construction (Self-Supervised Pretext Tasks)

To overcome the scarcity of paired 3D data, the authors propose automatic data engineering pipelines that convert generic 3D datasets (e.g., Objaverse-like collections) into training triplets $(z_H, z_L, I)$ without manual annotation. The specific augmentation strategies depend on the target task:

Compositional Enhancement: High-quality objects are arranged in grids and encoded into a single latent space. Noise is added to the scene tokens at a specific level $t^*$ and then denoised back to $t=0$ . The resulting "degraded" scene is decoded, and individual objects are extracted as low-quality pairs ( $y_L$ ) corresponding to the original high-quality objects ( $x_H$ ). This simulates the loss of detail common in compositional generation.
Guided Image-to-3D (Reconstruction): Ground-truth objects are rendered into 1–4 views. These views are fed into a feed-forward reconstruction network (VGGT) to produce sparse, incomplete point clouds ( $y_L$ ). These are aligned with the ground truth to form training pairs.
3D Shape Editing: A 3D mask is applied to a ground-truth object to remove a region. The remaining geometry is encoded, and the missing region is represented by learnable "mask tokens" initialized in the latent space. The guide image $I$ is generated by inpainting the masked region of a rendered view using a text-to-2D model.

Key Contributions

Unified Framework: MeshReGen introduces a single architecture capable of performing 3D enhancement, reconstruction from sparse views, and 3D editing, unifying tasks previously handled by disparate models.
Novel Conditioning Mechanism: The paper proposes a simple yet effective concatenation strategy for 3D conditioning within the VecSet latent space, outperforming cross-attention alternatives for dense or noisy inputs.
Scalable Data Protocols: The authors demonstrate protocols to automatically generate task-specific paired training data from generic 3D datasets via self-supervised augmentations, eliminating the need for expensive manual pairing.
State-of-the-Art Performance: The framework achieves superior results in geometric consistency and fine-grained detail recovery compared to task-specific baselines.

Results and Evaluation

The authors evaluate MeshReGen across three primary tasks:

Compositional 3D Enhancement: On decomposed scene objects, MeshReGen significantly outperforms baselines (including DetailGen3D) in geometric fidelity (ULIP-3D) and perceptual quality (MV-ImageReward). It successfully recovers details lost when generating large scenes in one pass.
Guided Image-to-3D (Reconstruction): Using the GSO dataset, MeshReGen combined with VGGT (VGGT+MeshReGen) achieves state-of-the-art reconstruction metrics (CD, F-score, IoU, PSNR). It notably outperforms single-view generators and matches or exceeds multi-view specialized models (like TRELLIS and MeshLRM) even when using fewer input views. It effectively fills in occluded regions while preserving visible geometry.
3D Shape Editing: Evaluated on the Edit3D-Bench dataset, MeshReGen demonstrates the ability to edit specific 3D regions while maintaining the identity and global structure of the original object. It avoids the artifacts and view inconsistencies common in 2D-to-3D lifting methods.

Ablation Studies:

Conditioning Mechanism: The proposed token concatenation with zero-initialized MLPs outperforms designs using separate cross-attention blocks (inspired by CLAY) or extended cross-attention (inspired by Hunyuan3D-Omni).
Diffusion Parameterization: The standard diffusion formulation outperforms rectified flow baselines in reconstruction tasks.
Generalization: Existing generation frameworks (TRELLIS, Hunyuan3D-Omni) fail to generalize to regeneration tasks with degraded inputs without fine-tuning, highlighting the necessity of the dedicated regeneration formulation and data.

Significance and Claims

The paper claims that MeshReGen represents a significant step toward unified, generalizable, and versatile 3D regeneration priors. By demonstrating that a single architecture can handle diverse downstream tasks (enhancement, reconstruction, editing) through a unified conditioning mechanism and self-supervised data construction, the work challenges the prevailing trend of task-specific 3D models.

The authors emphasize that their approach enables explicit control over global geometry (pose, coarse shape) while improving fine-grained details, bridging the gap between the flexibility of generative AI and the precision required by professional 3D workflows. They position MeshReGen not just as a generator, but as a tool for "regeneration" that can upgrade, repair, and modify existing 3D assets, making 3D content creation more accessible and efficient.