🤖 AI

VecSet-Edit: Unleashing Pre-trained LRM for Mesh Editing from Single Image

VecSet-Edit is the first pipeline to leverage pre-trained VecSet Large Reconstruction Models for direct 3D mesh editing from a single image, utilizing novel token-based localization, drift-aware pruning, and detail-preserving texture baking to overcome the limitations of existing voxel-based and multi-view approaches.

Original authors: Teng-Fang Hsiao, Bo-Kai Ruan, Yu-Lun Liu, Hong-Han Shuai

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Teng-Fang Hsiao, Bo-Kai Ruan, Yu-Lun Liu, Hong-Han Shuai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a 3D model of a cat, and you want to change its ears into watermelon slices without messing up the rest of its body. Or perhaps you want to turn a car's wheels into giant donuts while keeping the car's body exactly the same.

Doing this in the past was like trying to sculpt a statue out of a block of ice using a sledgehammer: you'd often break the parts you wanted to keep, or you'd have to spend hours manually carving out the specific area you wanted to change.

VecSet-Edit is a new tool that makes this process much smarter, faster, and more precise. Here is how it works, broken down into simple concepts:

1. The Problem: The "Unordered Bag of Marbles"

Most modern 3D tools that generate objects from a single picture use a method called VecSet. Think of a VecSet not as a solid 3D model, but as a bag of invisible marbles. Each marble represents a tiny piece of the object's shape and texture.

The problem is that these marbles are unordered. They are just floating in a bag. If you want to change the cat's ears, the computer doesn't inherently know which marbles belong to the ears and which belong to the tail. If you try to edit the "ears" by telling the computer to change "some marbles," you might accidentally change the tail too, or the whole cat might melt into a blob.

2. The Discovery: The Marbles Have "Memory"

The researchers discovered something amazing: even though the marbles are in a bag, they actually remember where they belong. If you look closely at how the computer "thinks" (its internal attention mechanisms), the marbles that form the ears consistently pay attention to the "ear" part of the 2D picture you gave them.

This is the core insight: You can find the right marbles just by looking at the 2D picture.

3. The Solution: A Three-Step Editing Process

VecSet-Edit uses a clever three-step process to edit the 3D model using only a 2D image and a simple mask (a drawing showing where you want to change things).

Step A: Finding the Right Marbles (Token Seeding & Gating)

The Analogy: Imagine you are looking for specific people in a crowded room to give a message to.
The Method:
1. Seeding: The system looks at your 2D drawing (the mask) and asks the computer, "Which marbles are looking at this specific part of the picture?" It picks a rough group of marbles that might be the ears.
2. Gating: This rough group might be too big or include the wrong marbles. The system then checks how these marbles talk to each other. If Marble A is talking to Marble B, and Marble B is definitely an ear, then Marble A is probably an ear too. This tightens the group so it only includes the exact marbles needed for the change.

Step B: The "RePaint" Dance (VecSet-Edit)

The Analogy: Imagine you are painting a mural, but you only want to repaint the sky, not the trees.
The Method: The computer starts with a noisy, blurry version of the 3D model. It slowly cleans up the noise (denoising).
- For the marbles you want to change (the ears), it listens to your new 2D image (the watermelon slices) and reshapes them.
- For the marbles you want to keep (the body), it ignores your new image and forces them to stay exactly where they were.
- This happens in a loop, slowly refining the shape until the ears look like watermelons, but the body is untouched.

Step C: The "Drift" Safety Net (Token Pruning)

The Analogy: Imagine you are rearranging furniture in a room. Sometimes, while moving a chair, you accidentally bump the table, and the table slides into the hallway.
The Method: Because the marbles are moving around during the editing process, sometimes a "body" marble might drift into the "ear" zone, or vice versa. This causes weird glitches.
- The system has a safety check halfway through the process. It looks for marbles that have drifted into the wrong neighborhood. If a "body" marble is hanging out in the "ear" zone, it gets kicked out (pruned) so it doesn't ruin the shape.

4. The Final Touch: Keeping the Texture Perfect

Once the shape is changed, the computer needs to paint the new surface.

The Problem: If you repaint the whole cat, the fur on the body might look blurry or different from the original.
The Solution: The system uses a "Detail-Preserving" technique. It only repaints the parts that actually changed shape (the watermelon ears). For the rest of the body, it simply copies the original, high-quality fur texture. This ensures the edited cat looks like a real cat with watermelon ears, not a blurry mess.

Why is this a big deal?

No 3D Masks Needed: You don't need to be a 3D artist to draw a mask in 3D space. You just draw on a 2D picture.
High Quality: Unlike older methods that used "voxels" (3D pixels, like a Minecraft block world), which make things look blocky, this method keeps the smooth, high-quality curves of the original model.
Identity Preserved: It's great at keeping the "soul" of the object. If you change a car's wheels, the car still looks like that specific car, just with new wheels.

In short, VecSet-Edit is like having a magic wand that lets you edit specific parts of a 3D object using a simple 2D drawing, while automatically knowing exactly which pieces to move and which to leave alone, ensuring the result looks professional and realistic.

1. Problem Statement

While automated 3D asset generation has advanced rapidly, localized 3D mesh editing remains a significant bottleneck. Existing approaches face several limitations:

Representation Issues: Most methods operate on intermediate representations like 3D Gaussian Splatting (3DGS) or Neural Radiance Fields (NeRF), which lack explicit topology required for animation and physics simulation.
Resolution and Fidelity: Voxel-based editing methods (e.g., VoxHammer) suffer from limited resolution and high computational costs at fine granularities.
Supervision Burden: Current mesh editing techniques often require labor-intensive 3D masks or bounding boxes to define edit regions, hindering scalability.
Identity Preservation: Regenerating assets based on text or image prompts often alters the identity of the original object or degrades fine details in unedited regions.

The authors propose a solution that enables localized, training-free 3D mesh editing guided solely by a single 2D image and a 2D binary mask, leveraging high-fidelity VecSet-based Large Reconstruction Models (LRMs).

2. Methodology

The proposed framework, VecSet-Edit, operates directly in the latent space of a pre-trained VecSet LRM (specifically TripoSG). The pipeline consists of four core components:

A. VecSet Geometry Property Analysis

The authors observe that although VecSet tokens are unordered, they exhibit spatial locality. A subset of tokens consistently corresponds to specific geometric regions of the mesh. This allows the editing problem to be reformulated as a token selection problem: identifying a subset of tokens ( $V_E$ ) responsible for the edit region while preserving the rest ( $V_P$ ).

B. Token Selection (Localization)

To map a 2D image mask to the correct 3D tokens without 3D supervision, the method employs a two-stage selection process:

Mask-guided Token Seeding:
- Utilizes cross-attention maps between VecSet tokens and the 2D image features.
- Aggregates attention scores over the masked region in the input image.
- Filters layers based on KL-divergence to select only the most informative attention layers, identifying a coarse set of seed tokens ( $V_I$ ) that align with the 2D mask.
Attention-aligned Token Gating:
- Expands the seed set using self-attention maps within the VecSet DiT (Diffusion Transformer).
- Since self-attention captures geometric proximity and topological adjacency, this step retrieves tokens strongly coupled to the seed set, ensuring the selected region is spatially coherent and covers the full 3D structure of the target object.

C. VecSet-Edit Framework (Diffusion & Pruning)

Once the editable token subset ( $V_E$ ) is identified, the framework performs constrained diffusion:

VecSet RePaint: Adapts the RePaint strategy to the latent token space. The editable tokens ( $V_E$ ) are iteratively denoised guided by the target edited image ( $I_E$ ), while the preserved tokens ( $V_P$ ) are constrained to follow their original diffusion trajectory to maintain identity.
Drift-aware Token Pruning: A critical innovation to handle the spatial mobility of VecSet tokens. During early denoising, editable tokens may "drift" into regions that should remain preserved, causing geometric overlaps. The method intervenes at a specific timestep ( $T_{pruning}$ $T_{p r u nin g}$ ) to:
- Identify tokens that have drifted into the preserved region ( $V_{conflict}$ ).
- Remove these conflicting tokens unless they are strictly necessary for the edit (supported by the target image condition).
- This ensures clean boundaries and prevents geometric artifacts.

D. Detail-Preserving Texture Baking

To update textures without blurring the original details:

The method computes a geometric difference mask between the source and edited meshes.
It uses an MV-Adapter (Multi-View Adapter) to generate multi-view textures.
Crucially, the texture generation is restricted only to the regions where geometry changed. This "in-painting" approach ensures that high-frequency details in unedited regions are perfectly preserved from the original mesh.

3. Key Contributions

First Training-Free VecSet Mesh Editing: Introduces the first framework to perform localized 3D mesh editing directly in the latent space of a high-fidelity VecSet LRM, avoiding the resolution bottlenecks of voxel-based methods.
Novel Token Selection Mechanisms: Proposes Mask-guided Token Seeding and Attention-aligned Token Gating to precisely localize edit regions using only 2D supervision, solving the challenge of mapping 2D masks to unordered 3D tokens.
Drift-aware Token Pruning: A unique strategy to remove geometric outliers during the diffusion process, ensuring structural consistency and preventing "bleeding" of edits into preserved areas.
Detail-Preserving Texture Baking: A texture synthesis module that updates only the modified geometry, maintaining the original asset's high-frequency appearance in unedited regions.

4. Experimental Results

The method was evaluated on the Edit3D-Bench dataset (300 samples) and compared against SOTA methods like VoxHammer, Trellis, MVEdit, and Instant3DiT.

Preservation Quality: VecSet-Edit achieved the lowest Chamfer Distance (CD) (0.011) among all editing methods, significantly outperforming VoxHammer (0.018) and Trellis (0.014). This indicates superior preservation of unedited geometry.
Visual Metrics: It achieved state-of-the-art PSNR (29.63) and SSIM (0.97) on unedited regions, demonstrating high fidelity.
Efficiency: The method is approximately 2x faster than Trellis and VoxHammer (approx. 200s vs. 600s) due to the efficiency of the VecSet backbone.
User Study: In a study with 12 3D practitioners, VecSet-Edit received 59.17% of votes for "Region Preservation" and 58.33% for "Condition Alignment," significantly outperforming all baselines.
Qualitative Results: Visual comparisons show that VecSet-Edit successfully handles complex topologies (e.g., hands, fans) and preserves intricate details that other methods often distort or lose.

5. Significance

VecSet-Edit bridges the gap between generative 3D models and controllable production workflows. By enabling precise, localized editing of explicit meshes using only a single image and a 2D mask, it removes the need for expensive 3D annotations and manual 3D masking. The ability to preserve asset identity and fine details while modifying specific regions makes this approach highly valuable for industries such as gaming, animation, and virtual reality, where generated assets often require refinement before use. The work also establishes a new paradigm for manipulating unordered latent representations in diffusion models.