PartSAM: A Scalable Promptable Part Segmentation Model Trained on Native 3D Data

PartSAM is the first promptable 3D part segmentation model trained natively on a large-scale dataset of over five million shape-part pairs, utilizing a triplane-based encoder-decoder architecture to achieve superior open-world generalization and accurate decomposition of both surface and internal structures compared to existing 2D-transfer methods.

The Gist

Imagine you have a giant, magical 3D printer that can spit out any object imaginable: a chair, a spaceship, a robot, or even a weird, AI-generated blob. Now, imagine you want to take that object apart, piece by piece, to paint the wheels red, replace the engine, or just understand how it's built.

For a long time, computers were terrible at this. They could see the object, but they couldn't "see" the parts. They were like a child looking at a car and only seeing "car," not "wheel," "door," or "engine."

Enter PartSAM, a new AI model introduced in this paper that changes the game. Think of PartSAM as a super-intelligent 3D surgeon who can look at any 3D object and instantly know exactly where one part ends and another begins.

Here is how it works, broken down into simple concepts:

1. The Old Way: "The 2D Photo Trick"

Previously, scientists tried to teach computers to understand 3D parts by showing them thousands of 2D photos from different angles. It's like trying to understand the inside of a house by only looking at photos of the outside walls.

• The Problem: The computer learns the surface but misses the inside. If you have a robot with a cloak, the computer sees the cloak but can't figure out what's underneath (the arms, the head). It also struggles with new, weird shapes it hasn't seen before.

2. The New Way: "PartSAM" (The Native 3D Brain)

PartSAM is different. Instead of learning from 2D photos, it was trained directly on millions of actual 3D models.

• The Analogy: Imagine learning to drive. The old way was reading a manual about cars (2D). PartSAM is like sitting in the driver's seat of a million different cars, feeling the steering wheel, the pedals, and the engine vibrations (Native 3D). It understands the structure of the object, not just the skin.

3. How It Learns: "The Model-in-the-Loop"

To teach PartSAM, the researchers needed a massive library of 3D objects with their parts labeled (e.g., "this is a leg," "this is a seat"). But manually labeling millions of 3D models is impossible for humans.

• The Solution: They built a self-teaching robot.
  1. They started with a smaller, decent model.
  2. They used that model to guess the parts of messy, unlabelled 3D shapes.
  3. Then, they used a second, smarter model to check those guesses. If the guess was good, they kept it. If it was bad, they threw it away.
  4. They repeated this loop until they had 5 million high-quality 3D examples. It's like a student grading their own homework, but with a very strict teacher making sure the answers are right.

4. How You Use It: "The Magic Click"

PartSAM is designed to be promptable, meaning you can talk to it or point at it.

• Interactive Mode: You click on a specific spot on a 3D chair (say, the backrest). PartSAM instantly highlights the entire backrest in red. You can click again to refine it. It's like playing a game of "spot the part."
• "Segment Every Part" Mode: You can tell PartSAM, "Just break this whole thing down for me." It will automatically separate the object into all its logical pieces (wheels, doors, seats) without you clicking anything.

5. Why It's a Big Deal

• It sees the invisible: Because it understands 3D geometry, it can figure out parts that are hidden or inside the object (like the steering wheel inside a car dashboard).
• It's flexible: It works on artist-made models, AI-generated blobs, and real-world scans. It doesn't care if the object is a chair or a dragon; it just knows how to break it down.
• It's fast: Unlike older methods that take minutes to process one object, PartSAM does it in seconds.

The Bottom Line

PartSAM is like giving a computer a pair of X-ray glasses and a deep understanding of how things are built. It moves us from just "seeing" 3D objects to truly "understanding" them, opening the door for better virtual reality, easier robot manipulation, and smarter 3D design tools. It's the first time a computer has learned to see 3D parts the way a human engineer does, directly from the 3D world itself.

Technical Summary

1. Problem Statement

Segmenting 3D objects into semantically meaningful parts is a fundamental challenge in computer vision and graphics. Existing approaches face three critical limitations:

• Closed-World Constraints: Traditional methods rely on small, predefined taxonomies (e.g., chairs have legs/seats), failing to generalize to unseen categories or open-world queries.
• Indirect Supervision (2D-to-3D Lifting): Recent "open-world" methods transfer knowledge from 2D foundation models (like SAM) by lifting multi-view 2D masks into 3D. This paradigm fails to capture intrinsic 3D geometry, leading to surface-only understanding, an inability to segment internal structures, and uncontrolled decomposition.
• Lack of Controllability & Scalability: Clustering-based 3D methods lack the user-centered controllability of promptable models, while existing feed-forward models often struggle with scalability and generalization due to data scarcity and reliance on 2D priors.

2. Methodology

The authors propose PartSAM, the first promptable 3D part segmentation model trained natively on large-scale 3D data. The framework consists of three core components:

A. Architecture: Dual-Branch Triplane Encoder-Decoder

• Encoder: Instead of standard point cloud networks, PartSAM encodes shapes into a continuous triplane feature field. It employs a dual-branch transformer:
  • Frozen Branch: Initialized with weights from PartField (trained via contrastive learning), this branch preserves powerful 2D priors derived from SAM.
  • Learnable Branch: Initialized similarly but fine-tuned on native 3D data. It accepts additional input attributes (normals, RGB) via a zero-convolution layer to enhance local detail representation.
  • The outputs are summed to create a robust, part-aware feature field.
• Prompt-Guided Decoder: Following the SAM design philosophy, the decoder takes input embeddings and user prompts (positive/negative points) to generate segmentation masks.
  • It uses a two-way transformer to attend bidirectionally between input patches and special tokens (output tokens for masks, an IoU token for quality estimation).
  • It supports interactive segmentation (iterative refinement) and automatic segmentation (generating multiple candidate masks).

B. Data Curation: Model-in-the-Loop Pipeline

To overcome the scarcity of high-quality 3D part annotations, the authors developed a scalable pipeline to curate over 5 million native 3D shape–part pairs:

1. Stage 1 (Direct Extraction): Extracts labels from artist-annotated scene graphs and connected components in datasets like Objaverse and PartNet. Filters out low-quality or overly fragmented meshes.
2. Stage 2 (Model-in-the-Loop): For highly fragmented assets where connected components do not form semantic parts, the pipeline uses a pre-trained PartSAM to interactively refine pseudo-labels generated by PartField.
   • PartField generates multi-scale clustering masks.
   • PartSAM simulates interactive segmentation (9 rounds) to filter these masks based on Intersection over Union (IoU) criteria.
   • Only shapes with >5 valid, high-confidence masks are retained, ensuring diverse and fine-grained supervision.

C. Training Strategy

• Simulation: Simulates interactive segmentation by sampling prompts from ground-truth centers and error regions.
• Loss Functions: Combines Focal Loss, Dice Loss, and an IoU prediction loss. A triplet contrastive loss is used to strengthen the encoder's representation.

3. Key Contributions

1. First Native 3D Foundation Model: PartSAM is the first scalable, feed-forward, promptable model trained directly on millions of native 3D shape-part pairs, bypassing the limitations of 2D lifting.
2. Dual-Branch Encoder Design: A novel architecture that simultaneously retains 2D priors (from SAM) and learns fine-grained 3D semantics, enabling effective scaling to native 3D supervision.
3. Scalable Annotation Pipeline: A model-in-the-loop strategy that expands training data to 500k shapes and 5M parts, providing diverse supervision for open-world scenarios.
4. "Segment Every Part" Mode: An emergent capability allowing the model to automatically decompose entire shapes into both surface and internal structures without user prompts.

4. Experimental Results

PartSAM was evaluated on multiple benchmarks (PartObjaverse-Tiny, PartNet-E, 3DCoMPaT++) and compared against SOTA methods like Point-SAM, SAMesh, SAMPart3D, and PartField.

• Interactive Segmentation:
  • Outperforms Point-SAM by a massive margin. On PartObjaverse-Tiny, it achieves 56.1% IoU with a single prompt (vs. 29.4% for Point-SAM), a 91% relative improvement.
  • Demonstrates superior ability to segment occluded and internal parts (e.g., the knee of a leg, seats inside a car) which 2D-lifting methods miss.
• Automatic Segmentation:
  • Achieves 69.5% IoU on PartObjaverse-Tiny and 72.4% on PartNet-E, surpassing the second-best method by over 20%.
  • Successfully segments AI-generated meshes (Hunyuan3D) and complex structures where clustering-based methods fail or produce fragmented results.
• Generalization:
  • Shows strong performance on the fine-grained 3DCoMPaT++ dataset, handling rare and small parts better than clustering-based baselines.
  • Demonstrates robust cross-shape consistency in functional map evaluations.

5. Significance

PartSAM marks a decisive step toward foundation models for 3D part understanding. By shifting from indirect 2D supervision to native 3D training, the model achieves:

• True 3D Awareness: The ability to understand and segment internal structures and occluded parts, which is impossible for 2D-lifting approaches.
• Controllability: Flexible interaction via single prompts or automatic decomposition, bridging the gap between rigid taxonomy-based models and open-world needs.
• Scalability: Proves that 3D vision models can scale effectively with large-scale native data, opening new avenues for 3D asset creation, AR/VR editing, and robotic manipulation.

The work establishes that combining a scalable architecture with diverse, large-scale native 3D data is the key to unlocking generalizable, geometry-aware 3D perception.