🤖 AI

PartMat: Material-Aware 3D Part Decomposition with a Single Global Latent

PartMat is an efficient, material-aware 3D part decomposition pipeline that utilizes a single global latent representation to generate structured, editable assets with accurate material boundaries and fine-grained geometry while decoupling inference costs from the number of parts.

Original authors: Guangming Fu, Jin Song, Yiyun Fei, Guoqiu Li, Ruigao Yang, Jianan Jiang

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guangming Fu, Jin Song, Yiyun Fei, Guoqiu Li, Ruigao Yang, Jianan Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a digital world, like in your favorite video game or a movie. For a long time, computers were great at making the shape of things—a chair, a table, or a robot—but they treated the whole object as one giant, unbreakable lump of digital clay. If you wanted to change the chair's fabric to leather or swap the wooden legs for metal, you were stuck; the computer didn't know where one material ended and another began. Recently, scientists have figured out how to break these digital objects into pieces, like taking a real chair apart into its legs, seat, and back. But there's a catch: most of these new methods only know how to separate things by function (like "this is the leg") rather than by material (like "this is the wood part" or "this is the metal part"). In the real world, a single chair leg might be made of wood, but the top might be wrapped in fabric, or a table might have a glass top and a metal base. To make truly editable 3D worlds where you can swap out materials easily, computers need to understand these material boundaries, not just the structural ones.

This is where a new tool called PartMat comes in. Think of it as a super-smart digital chef who can look at a whole, cooked meal and instantly separate it into its exact ingredients—distinguishing the crispy bacon from the soft eggs and the crunchy toast—without messing up the shape of the food. The researchers behind PartMat noticed that previous methods were either too slow (taking forever to separate every single piece) or too clumsy (mixing up the materials). They built a system that can take a single image and a rough 3D shape, then instantly chop it up into perfectly separated material parts, like "brushed brass," "painted wood," or "white marble," ready for you to edit.

The magic of PartMat lies in how it thinks. Instead of trying to build every single piece of the puzzle one by one—which is like trying to paint a whole mural by painting one tiny dot at a time, over and over again—they use a "global brain." Imagine a single, compact secret code (a "global latent") that holds the blueprint for the entire object's materials at once. When the computer needs to show you the parts, it reads this one code and spits out all the pieces simultaneously. This means it doesn't matter if the object has 2 parts or 32 parts; the computer takes the same amount of time to figure it out. It's like having a master key that opens every door in a castle instantly, rather than needing a different key for each one.

To make sure the pieces fit together perfectly and don't accidentally overlap (like trying to put a wooden leg inside a metal leg), the system uses a clever training trick called "reinforcement learning." Think of this like a video game where the computer gets a "reward" for doing a good job and a "penalty" for making mistakes. If the computer tries to make two parts occupy the same space, it gets a digital "ouch" and learns to stop. If it correctly matches the material to the shape in the reference image, it gets a "high five." This helps the system learn to draw the lines between materials with extreme precision.

Finally, because the initial "secret code" is so compact, the edges of the materials might look a little blurry, like a low-resolution photo. To fix this, the team added a "refinement stage." This is like taking that blurry photo and running it through a high-definition filter that sharpens the edges, making the wood grain look real and the metal shine. The result is a set of 3D parts that are not only separated by material but also look incredibly detailed and realistic.

The researchers tested this on about 300,000 different furniture and household objects. They found that PartMat is much better at separating materials than previous methods, achieving higher accuracy in matching the right material to the right spot. It also keeps the 3D shapes looking just as good as the best existing tools, but without the slow speed. While the system currently works best when the materials are clearly defined and can handle up to 32 different parts at once, it represents a significant step forward. It suggests that we are getting closer to a future where creating and editing 3D worlds is as easy as swapping out the wallpaper in a room, with the computer understanding exactly where the wood ends and the fabric begins.

Technical Summary: PartMat: Material-Aware 3D Part Decomposition with a Single Global Latent

Problem Definition

Current 3D generation and decomposition methods primarily focus on functional semantics (e.g., chair legs, seats) or geometric structure. However, practical applications such as interior design and embodied simulation require decomposition based on editable material boundaries (e.g., separating fabric, wood, metal, and glass regions). Existing approaches face two critical limitations:

Inefficiency: Methods that generate parts independently (e.g., PartCrafter, PartPacker) suffer from computational costs that scale linearly with the number of parts, making them inefficient for objects with many material regions.
Inconsistency: Separate per-component representations often weaken global coordination, leading to issues such as misassigned material regions, disconnected components of the same material being split, and overlapping or duplicated parts. Furthermore, cascaded pipelines that first generate textures and then segment them propagate errors from appearance generation to geometric decomposition.

The goal of this work is material-aware 3D part decomposition: given a reference image and a single whole-object geometry mesh (without part annotations), decompose the object into parts whose boundaries strictly follow editable material assignments, enabling independent editing and property assignment.

Methodology

PartMat is a three-stage framework designed to represent multi-part geometry with a single global latent, decoupling inference cost from the number of parts.

Stage I: PartVAE (Unified Representation)

The authors introduce PartVAE, a Variational Autoencoder that learns a unified representation for multi-component geometry.

Architecture: It adapts the Hunyuan3D 2.1 ShapeVAE architecture. A Part-aware Point Encoder processes surface points (position, normal, material ID, sharp-edge indicator) and aggregates them via cross-attention into a single global latent vector ( $z$ ).
Decoding: Unlike methods that decode parts sequentially, PartVAE employs a Multi-channel SDF Decoder. It takes the single global latent $z$ and jointly decodes $K$ (e.g., 32) Signed Distance Field (SDF) channels in a single forward pass. Each channel corresponds to a potential material part.
Efficiency: Because the latent size and the number of decoder passes are fixed regardless of the actual number of active parts ( $M \le K$ ), the decoding latency is constant and independent of the part count.
Training Objective: The model is trained with a reconstruction loss for valid parts and a suppression loss to ensure non-existent parts remain away from the zero level set.

Stage II: PartDiT with RL Alignment (Generation)

A conditional diffusion model, PartDiT, generates the global latent from a reference image and the input whole-object geometry.

Architecture: Based on the Hunyuan3D 2.1 DiT, it uses separate cross-attention layers to attend to image tokens (from DINOv2) and geometry tokens (from the frozen PartVAE encoder).
RL Alignment: To address the issue of overlapping parts and inaccurate material assignment, the authors employ Direct-Gradient Reinforcement Learning (RL). Instead of decoding every sample to a mesh for evaluation (which is costly), they compute a differentiable reward directly in the implicit SDF space of the frozen PartVAE.
- Matching Reward ( $R_{match}$ ): Encourages predicted channels to align with ground-truth material regions using soft occupancy similarity and differentiable assignment.
- Overlap Penalty ( $R_{overlap}$ ): Penalizes simultaneous occupancy by different predicted components.
- The model is fine-tuned using a loss function that maximizes this differentiable reward, effectively suppressing overlaps and refining material boundaries.

Stage III: Part-Aware Sparse Geometry Refinement

While PartVAE provides coherent geometry, the compact latent may smooth thin structures.

Approach: A high-resolution sparse-voxel flow-matching model refines the coarse components.
Mechanism: The refinement network is conditioned on the coarse parts, the global geometry, and the reference image. Crucially, it introduces Part Attention, which interleaves full self-attention with within-part attention layers. This preserves the identity of each material slot while allowing communication across material interfaces, ensuring seamless boundaries and recovering high-frequency geometric details.

Key Contributions

PartVAE: The first VAE to represent multi-component geometry with a single global latent, successfully decoupling decoding latency from the number of parts.
PartMat Framework: A three-stage pipeline that integrates:
- Direct-gradient reinforcement learning for precise material decomposition and overlap suppression.
- A part-aware sparse flow refiner to restore high-resolution geometry with seamless component boundaries.
Performance: Extensive experiments on ~300K material-annotated objects demonstrate state-of-the-art material decomposition accuracy (Sem-IoU) and comparable geometric quality to existing baselines, while maintaining efficient inference.

Experimental Results

The authors evaluated PartMat on a dataset of approximately 300K furniture and household objects.

VAE Reconstruction: PartVAE achieves competitive geometric reconstruction (CD and F1 scores) compared to single-object VAEs (Hunyuan2.1, TRELLIS2) while providing editable material channels. Unlike PartCrafter, its decode latency remains nearly constant (e.g., ~1.7s for 32 components) rather than scaling linearly.
Image-to-3D Generation: In image-conditioned generation, PartMat significantly outperforms baselines (X-Part, PartCrafter, PartPacker, HoloPart, OmniPart) in material decomposition accuracy (Sem-IoU: 50.51% with RL vs. 43.00% for X-Part) and geometric fidelity (CD: 3.67 vs. 5.13).
Ablation Studies:
- Representation: The global multi-channel strategy outperforms local-box (which struggles with thin structures) and packed-sem (which creates jagged boundaries) approaches.
- RL Alignment: The combination of matching and overlap rewards yields the closest decomposition to ground truth, reducing spatial conflicts and blurred boundaries.
- Refinement: The sparse flow refiner successfully recovers sharp edges and micro-structures lost in the compact latent.

Significance and Claims

The paper claims that PartMat addresses a critical gap in 3D asset creation: the ability to generate structured, material-editable assets efficiently. By moving away from function-based semantics to material boundaries, the method enables downstream applications such as interior design (replacing specific materials like "painted wood" or "brushed brass") and physics simulation (assigning distinct friction or density properties to specific regions).

The authors emphasize that their approach uniquely maintains constant decoding latency regardless of part complexity, a significant advantage over autoregressive or per-part generation methods. They position PartMat as a foundational step for creating 3D assets that are not only geometrically accurate but also semantically aligned with the physical materials required for realistic rendering and interaction.

Limitations Acknowledged:
The method assumes a fixed component capacity ( $K=32$ ) and relies on high-quality material annotations; weakly supervised alternatives may introduce boundary noise. Additionally, reconstructing extremely tiny details remains challenging and requires further refinement.