🤖 machine learning

CompoSE: Compositional Synthesis and Editing of 3D Shapes via Part-Aware Control

CompoSE is a novel diffusion transformer-based method that synthesizes and edits high-quality 3D shapes by converting coarse geometric part layouts into detailed, part-separated objects with granular editing capabilities, all without requiring part-level text prompts.

Original authors: Habib Slim, Shariq Farooq Bhat, Mohamed Elhoseiny, Yifan Wang, Mike Roberts

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Habib Slim, Shariq Farooq Bhat, Mohamed Elhoseiny, Yifan Wang, Mike Roberts

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to build a custom 3D chair, but instead of sculpting it from a single block of clay, you want to design it by placing specific "zones" for the legs, the seat, and the backrest. You want to tell the computer, "This box is for a leg, this one is for the seat," and then have the computer fill in the details.

That is exactly what CompoSE does. It is a new tool that lets you create and edit 3D objects by arranging simple shapes (like floating boxes) and giving a text description, rather than trying to describe the whole complex object with just words.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Magic Box" is Too Unpredictable

Usually, when you ask an AI to "make a chair," it acts like a magic box. You give it a sentence, and it spits out a chair. But if you want to change just the legs to be longer, or swap the seat for a cushion, you often have to start over from scratch. The AI doesn't understand that a chair is made of separate parts; it sees it as one giant blob. If you change the text slightly, the whole chair might change shape, color, or style in unexpected ways.

2. The Solution: The "Architect's Blueprint"

CompoSE changes the game by letting you act like an architect.

The Input: You don't just type "chair." You also draw a few floating boxes in 3D space. One box for the legs, one for the seat, one for the back.
The Magic: The AI looks at your boxes and your text (e.g., "a wooden dining chair") and fills in the details inside those specific boxes.
The Result: You get a chair where the legs are exactly where you put the leg-box, and the seat is exactly where you put the seat-box.

3. The Secret Sauce: The "Local vs. Global" Brain

How does the AI know that the leg-box should look like a leg and not a table leg? It uses a special brain architecture (a Diffusion Transformer) that thinks in two ways at once:

Local Thinking: It looks at one box at a time. "Okay, this box is small and low; it needs to be a leg." It focuses on the specific shape of that part.
Global Thinking: It looks at the whole picture. "Wait, if these are legs and this is a seat, they need to match in style and size so it looks like a real chair, not a pile of mismatched wood."

By switching back and forth between looking at the details and looking at the big picture, it ensures every part fits together perfectly.

4. The Editing Superpower: "Swap and Keep"

The coolest part of CompoSE is how easy it is to edit. Because the AI understands the object as separate parts, you can do things like:

Resize without Ruining: You can stretch the box for the chair back to make it taller. The AI stretches the backrest but keeps the wood grain, texture, and style exactly the same. It doesn't turn the chair into a different object; it just makes the part bigger.
Swap Parts: You can delete the box for the seat and draw a new one, then tell the AI, "Make this a leather cushion." The legs and back stay exactly as they were; only the seat changes.
Add Parts: You can add a new box for an armrest, and the AI will grow an armrest that matches the rest of the chair.

5. How They Trained It: The "Auto-Labeling" Factory

To teach the AI this, the researchers needed thousands of examples of 3D objects broken into parts. Since no one had a database of "chairs with labeled legs and seats," they built a robot pipeline.

They took thousands of raw 3D models from a massive library called Objaverse.
They used a smart algorithm to automatically chop the models into logical pieces (like separating a table top from its legs).
They used a vision-language AI to look at the object and write a description for it.
This created a massive training set without any humans having to manually label every single part.

Summary

Think of CompoSE as a Lego set for 3D design. Instead of trying to mold a whole statue out of clay (which is hard to fix if you mess up), you place your "Lego bricks" (the boxes) where you want them. The AI then fills in the texture and shape of each brick individually, ensuring they all snap together to form a beautiful, coherent object that you can tweak piece by piece.

What it does NOT do (based on the paper):

It does not guarantee the object is physically stable (a chair might look good but fall over if you put weight on it).
It does not currently work with more than 8 parts at a time.
It does not automatically guess where to put the boxes; you still have to place them yourself (though the paper suggests future work could do this).

Technical Summary: CompoSE

Problem Statement

Creating and editing high-quality 3D content remains a central challenge in computer graphics. While recent generative AI methods can synthesize 3D shapes from high-level inputs (text, images, or global bounding boxes), they lack compositional synthesis and editing capabilities. Existing approaches typically model user intent through a single monolithic object or rely on 2D image conditioning, which entangles part geometry and prevents localized, granular editing. Specifically, current methods cannot accept part-separated inputs (e.g., distinct bounding boxes for individual object parts) to guide synthesis, nor do they support editing individual parts (substitution, resizing, deletion) while preserving the identity and style of the rest of the object. Furthermore, no existing method supports inferring part semantics and symmetries directly from coarse layout guidance without requiring part-level text prompts.

Methodology: CompoSE

The authors introduce CompoSE (Compositional Synthesis and Editing of 3D Shapes via Part-Aware Control), a novel method that synthesizes part-separated 3D objects from a set of coarse geometric primitives (e.g., oriented bounding boxes) and a global text prompt.

Core Architecture

The method utilizes a Diffusion Transformer (DiT) architecture designed to alternate between local and global processing:

Local Blocks: Process spatial layout information for individual primitives independently. This provides high-fidelity control over the geometry of each synthesized part, conditioned on the specific input box and the shared text embedding.
Global Blocks: Aggregate contextual cues and text information across all parts. This ensures stylistic coherence, structural validity, and harmonization of global proportions across the entire object.
Conditioning Mechanism: The model fuses part-level geometric conditions with a global text prompt. It learns to infer how global semantics distribute across parts and how parts interact to form a coherent whole, without requiring text prompts for individual parts.

Representations and Encoding

3D Shape Representation: The model operates in the latent space of a pre-trained VAE (TripoSG). Ground truth 3D parts are mapped to latent Signed Distance Function (SDF) representations.
Input Geometry Encoding: Instead of encoding bounding box parameters (center, extent, orientation) via a simple MLP, CompoSE encodes each input box using the same SDF-based shape encoder used for 3D parts. Points and normals are sampled from the box surface to generate a latent embedding. This ensures latent-space alignment between the conditioning signal (boxes) and the target geometry, significantly improving adherence to the input layout.

Editing Capabilities

CompoSE enables localized granular editing through two primary techniques:

Part-Latent Freezing: To edit a subset of parts while preserving others, the latent variables of unedited parts are frozen by restoring them from the reference shape at every diffusion time step. This locks unedited parts in place, maintaining their geometry and semantics.
KV Reinjection (Identity-Preserving Resizing): For geometric adjustments (e.g., resizing) where the style must be preserved, the method caches attention keys and values from the reference shape. During editing, these cached values are blended with recomputed values from the modified latents in a time-step-dependent manner. This anchors the visual identity, ensuring that geometric changes do not alter the part's style or texture.

Data Pipeline

Since no existing dataset contains part-segmented shapes with corresponding text-layout pairs, the authors developed a fully automated data processing pipeline:

Segmentation: Raw 3D meshes are decomposed into low-level segments, which are progressively merged based on volume heuristics to create semantically meaningful parts.
Layout Extraction: Oriented Bounding Boxes (OBBs) are computed for each part segment.
Caption Generation: A vision-language model (InternVL3-8B) generates global text prompts from rendered multi-view images, focusing on geometry and part descriptions while avoiding irrelevant color/texture details.

Key Contributions

First Part-Separated Synthesis: The first method to synthesize part-separated 3D shapes from coarse part layouts, featuring a novel conditioning technique that ensures strong adherence to user input layouts and high control over individual parts.
Versatile Compositional Editing: Enables a set of editing capabilities including substitution, addition, deletion, and style-preserving resizing of individual object parts.
Automated Data Pipeline: A fully automated pipeline enabling training on large-scale 3D shape datasets without requiring manual part-level segmentations or part-level text annotations.

Results and Evaluation

The authors evaluated CompoSE on unseen part layouts and prompts from the Objaverse dataset, comparing it against baselines such as Fantasia3D, Coin3D, Spice-E, and SpaceControl.

Quantitative Metrics: CompoSE significantly outperforms existing approaches in guided synthesis. The Ours2048 variant achieved the best Object-FID (12.77) and Part-FID (14.41), indicating superior visual quality. It also achieved the highest Part-IoU (0.840), demonstrating strong alignment with input layouts, and the best CLIP-Score (31.07) among larger models.
LLM-Based Evaluation: Using GPT-4o as a judge for layout adherence, prompt adherence, quality, and geometry, CompoSE achieved the best mean rank (1.69) across all axes, substantially outperforming all baselines.
Editing Performance: In identity-preserving rescaling tasks, CompoSE achieved significantly higher F-Scores (0.70 at $\tau=0.05$ ) compared to anisotropic scaling (0.55) and part substitution (0.64), confirming its ability to maintain geometric identity during edits.
Inference Time: The method scales gracefully with the number of parts ( $N_p$ ). At 512 resolution, it is faster than optimization-based baselines even with 8 parts.

Significance and Limitations

The paper claims that CompoSE addresses a critical gap in professional content creation by enabling compositional synthesis and editing, allowing users to manipulate individual parts while maintaining global coherence. The authors suggest that providing generative AI with rich structured guidance (like part layouts) could become a standard in the computer graphics toolbox, potentially enabling the synthesis of simulation-ready assets from coarse proxy meshes.

Limitations acknowledged by the authors include:

Artifacts: Slight artifacts (e.g., z-fighting) can occur at part boundaries due to the VAE's out-of-distribution nature regarding part segments.
Physical Plausibility: The method does not guarantee physical stability (e.g., missing contacts or asymmetries).
Scalability: The current architecture is capped at eight parts per shape due to the computational cost of global attention blocks.
User Input: Users must still manually define input bounding boxes; the authors note that a text-to-structure model could automate this in the future.