Vinedresser3D: Agentic Text-guided 3D Editing

Vinedresser3D is an agentic framework that leverages a multimodal large language model and an image editing model to decompose complex text prompts and perform precise, mask-free 3D editing directly in the latent space of a native 3D generative model, ensuring high-quality prompt alignment while preserving the coherence of unedited regions.

The Gist

Imagine you have a magical, digital LEGO set. In the past, if you wanted to turn a toy car into a train, you had to be a master builder. You'd have to manually take apart every single brick, swap them out, and hope the rest of the model didn't fall apart. It was slow, required special skills, and was frustratingly difficult.

Vinedresser3D is like hiring a super-smart, robotic gardener who can instantly reshape your digital garden based on a simple sentence you say.

Here is how it works, broken down into simple steps:

1. The "Brain" (The Agent)

Think of Vinedresser3D as a digital butler with a very powerful brain (an AI called an MLLM).

• You speak: You say, "Turn that toy car into a train."
• The Brain thinks: Instead of just blindly changing pixels, the butler looks at your 3D object, understands what a "car" is and what a "train" is, and figures out exactly which parts need to change. It realizes, "Okay, I need to change the body and the wheels, but I must keep the little duck sitting on top exactly where it is."
• The Plan: It writes a detailed recipe for the change, describing the new train body and wheels, while promising to leave the duck alone.

2. The "Eyes" (Finding the Spot)

In the old days, you had to draw a circle around the part you wanted to change (a "mask"). Vinedresser3D doesn't need you to do that.

• The Magic Sight: The butler uses a special pair of glasses (a 3D segmentation tool) to look at the object and automatically figure out where the "car body" ends and the "duck" begins. It draws an invisible line around the car parts so it knows exactly what to touch and what to leave untouched.

3. The "Hands" (The Editing)

Now comes the actual magic trick. The system uses a technique called "Inversion-Based Editing."

• The Analogy: Imagine you have a sculpture made of clay. Instead of chipping away at it, the system first turns the whole sculpture back into a cloud of dust (noise) that still holds the memory of the original shape.
• The Remix: It then takes that cloud of dust and starts rebuilding it. But here's the trick: it only rebuilds the parts that are inside the "invisible line" (the car). The parts outside the line (the duck) are kept safe in their original form.
• The Double-Check: To make sure the new train looks perfect, the system uses two different experts: one who is great at reading text descriptions (to get the idea right) and one who is great at looking at pictures (to get the shiny details right). It switches back and forth between them, like a chef tasting a soup and adjusting the spices, until the train looks perfect.

Why is this a big deal?

• No More "Global" Mistakes: Old methods often changed the whole object when you only wanted to change the wheels. Vinedresser3D is like a surgeon; it only operates on the specific spot you asked for.
• No More Drawing: You don't need to be an artist to draw a mask around the object. You just talk to the AI.
• It Keeps the Good Stuff: If you have a complex scene with a cart full of watermelons, and you ask to "add a billboard," the system adds the billboard but keeps the watermelons looking exactly the same. It doesn't accidentally turn the watermelons into rocks.

The Result

The paper shows that this "Agentic" approach (where the AI acts like a smart assistant rather than just a tool) creates 3D edits that are:

1. Accurate: It does exactly what you asked.
2. Clean: It doesn't break the parts you didn't ask to change.
3. High Quality: The new objects look like they belong in the scene, not like a blurry sticker pasted on top.

In short, Vinedresser3D turns the difficult, technical job of 3D editing into a simple conversation, making it possible for anyone to reshape their digital world just by speaking their mind.

Technical Summary

1. Problem Statement

Text-guided 3D editing aims to modify existing 3D assets using natural language instructions. Despite advancements in 3D generation, current editing methods face three critical challenges:

• Semantic Understanding: Difficulty in jointly understanding complex, multi-part editing prompts.
• Automatic Localization: Inability to automatically identify precise 3D editing regions from text alone, often requiring manual 3D masks.
• Preservation: Struggles to preserve unedited content (geometry and appearance) while applying changes, leading to unintended global deformations or loss of detail.

Existing approaches generally fall into two categories with limitations:

1. Score Distillation Sampling (SDS): Computationally expensive per-scene optimization that often causes unintended global changes.
2. 2D Editing + 3D Reconstruction: Suffers from multi-view inconsistency and incomplete 3D supervision, failing to preserve unobserved regions.
3. Native 3D Editing (e.g., VoxHammer): Operates in 3D latent space but still relies on user-provided 3D masks and cannot handle complex edit requests.

2. Methodology: Vinedresser3D

The authors propose Vinedresser3D, an agentic framework that operates directly in the latent space of a native 3D generative model (Trellis). The system integrates a Multimodal Large Language Model (MLLM) with specialized 3D tools to perform high-quality, mask-free editing.

The pipeline consists of four main stages:

A. Multi-modal Guidance Generation (The "Brain")

• Input: A 3D asset (rendered as multi-view images) and a user text prompt.
• MLLM Core: Uses Gemini-2.5-flash to semantically parse the instruction.
• Output:
  1. Original Description: A detailed description of the input asset.
  2. Edit Classification: Identifies the edit type (Addition, Modification, Deletion) and specific parts to target.
  3. New Descriptions: Generates a complete description of the post-edit asset and isolates descriptions for the new parts.
  4. Decomposition: Splits descriptions into Stage 1 (structure/geometry) and Stage 2 (appearance/texture) to align with the two-stage Trellis generation process.
• Visual Guidance: The MLLM selects the optimal view for visibility and uses an image editing model (Nano Banana) to generate a high-fidelity reference image based on the prompt and decomposed text.

B. Automatic Editing Region Detection

• Goal: Eliminate the need for user-provided 3D masks.
• Process:
  1. Segmentation: Uses PartField to decompose the 3D asset into semantic parts.
  2. Selection: The MLLM analyzes the segmented parts against the text prompt to select the specific parts to edit ($P_{edit}$) and preserve ($P_{pres}$).
  3. Voxel Mapping: Converts semantic parts into voxel masks ($R_{edit}$).
     • For Addition/Deletion: Logic is straightforward (outside asset or inside asset).
     • For Modification: Uses a K-Nearest Neighbor (KNN) approach. It calculates the proportion of neighboring voxels belonging to the edit region. If this proportion exceeds a threshold $\tau$, the voxel is marked as editable. This prevents "bleeding" into preserved geometry.

C. Inversion-Based 3D Editing (The "Engine")

• Base Model: Uses Trellis, a flow-based native 3D generative model using Structured Latent (SLAT) representations.
• Inversion: Uses RF-Solver (a second-order Taylor expansion method) to invert the original 3D asset back into structured noise in the latent space, conditioned on the original asset description.
• Interleaved Inpainting:
  • Instead of using only text or only image conditions, the agent performs interleaved denoising.
  • It alternates between Trellis-text (for semantic alignment and prompt adherence) and Trellis-image (for high-fidelity visual details) at each timestep.
  • Mask-Guided Inpainting: During the denoising process, voxels outside the editing region ($R_{edit}$) are replaced with features from the original inversion trajectory to ensure preservation.
  • Soft Masking: For voxels near the boundary of preserved regions, features are computed as a weighted average to eliminate floating artifacts.

3. Key Contributions

1. Agentic Framework: Introduces the first 3D editing agent that uses an MLLM to coordinate text understanding, region detection, and tool invocation without manual 3D masks.
2. MLLM Integration: Demonstrates that MLLMs (trained primarily on 2D data) can effectively plan 3D editing strategies, generate multi-modal guidance, and interact with 3D segmentation and generation tools.
3. Mask-Free Precision: Develops a novel pipeline for automatically detecting editing regions using 3D segmentation and spatial reasoning, removing the bottleneck of manual mask creation.
4. Interleaved Editing: Proposes an interleaved sampling module that combines text and image conditioning to overcome the limitations of single-modality editing (text lacks detail; image lacks occlusion handling).

4. Experimental Results

The method was evaluated on a diverse dataset of 3D assets (objects and scenes) against baselines: Trellis (with masks), VoxHammer, and Instant3dit.

• Quantitative Metrics:
  • Text Alignment: Achieved the highest CLIP-T score (0.252), outperforming all baselines.
  • Preservation: While slightly lower than Trellis with human masks on preservation metrics (PSNR, SSIM), it significantly outperforms other mask-free or semi-automatic methods.
  • Quality: Achieved the best FID scores (29.49), indicating superior overall 3D quality.
• Human Preference Study:
  • Users preferred Vinedresser3D over Trellis and VoxHammer in 92.5% and 89.8% of cases, respectively, across text alignment, preservation, and overall quality.
• Ablation Studies:
  • Removing the Interleaved design (using only Trellis-image) led to distorted outputs and lower FID.
  • Removing the Editing Region Mask resulted in the alteration of preserved parts and distorted geometry, confirming the necessity of the detection module.

5. Significance and Future Work

• Significance: Vinedresser3D bridges the gap between high-level natural language instructions and precise 3D manipulation. It moves the field from "optimization-heavy" or "mask-dependent" workflows to an agentic, automated, and high-fidelity paradigm. It proves that 2D-trained MLLMs can be successfully integrated into 3D pipelines to solve complex spatial reasoning tasks.
• Limitations: The current MLLM does not accept native 3D input (relies on rendered views), and the performance is dependent on the quality of external tools like PartField for segmentation.
• Future Direction: The authors suggest that enabling MLLMs to directly consume 3D inputs and improving 3D segmentation models will further enhance the system's capabilities.

In conclusion, Vinedresser3D represents a significant step toward democratizing 3D content creation, allowing non-experts to perform complex, precise edits on 3D assets using simple text prompts.