💻 computer science

CAD-Prompted SAM3: Geometry-Conditioned Instance Segmentation for Industrial Objects

This paper proposes CAD-Prompted SAM3, a geometry-conditioned instance segmentation framework that leverages multi-view CAD renderings as prompts to overcome the limitations of language and appearance-based methods in industrial settings where objects vary in material and finish but share canonical geometry.

Original authors: Zhenran Tang, Rohan Nagabhirava, Changliu Liu

Published 2026-03-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhenran Tang, Rohan Nagabhirava, Changliu Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a robot working in a busy factory. Your job is to pick up specific parts from a messy pile of tools and materials. To do this, you need to know exactly what to grab.

In the past, robots had two main ways to figure this out, and both had big problems:

The "Talker" Problem: You could tell the robot, "Grab the red screw." But what if the screw is blue today? Or what if it's a weird, custom-shaped part that doesn't have a name like "screw" or "bolt"? The robot gets confused because it relies on words and colors.
The "Show-and-Tell" Problem: You could show the robot a photo of the part you want. But in a factory, the same part might be made of shiny metal one day and matte plastic the next. If you show the robot a photo of the shiny version, it might get confused when it sees the matte version. It's too focused on the look (texture and color) rather than the shape.

The New Solution: The "Blueprint" Robot

This paper introduces a new way to teach robots called CAD-Prompted SAM3. Instead of using words or photos, it uses the digital blueprints (CAD models) that engineers use to design the parts.

Here is how it works, using a simple analogy:

1. The Blueprint vs. The Costume

Think of a factory part like a character in a play.

The Appearance (Costume): The part might wear a red coat, a blue hat, or be made of gold. This changes all the time.
The Geometry (The Actor): The actual shape of the part is the actor underneath. It never changes, no matter what costume they wear.

Old methods tried to recognize the costume. If the actor changed clothes, the robot didn't recognize them.
This new method gives the robot the blueprint of the actor. It says, "Ignore the clothes. Look for this specific shape."

2. How the Robot "Sees" the Blueprint

Computers can't look at a 3D blueprint file directly the way a human looks at a drawing. So, the researchers invented a clever trick:

The "Magic Camera": They take the 3D blueprint and spin it around, taking pictures of it from 12 different angles (top, bottom, sides).
The "Shape Translator": They feed these pictures into a super-smart AI (called SAM3). This AI learns to ignore the colors in the blueprint pictures and focuses entirely on the edges and curves.
The Match: When the robot looks at the real, messy factory floor, it compares the "shape-only" blueprint pictures against the real scene. It finds the object that matches the shape, even if the real object is a different color or covered in grease.

3. Why This is a Game-Changer

Imagine you are building a custom Lego set.

Old Way: You have to find a photo of the exact brick you need. If the photo is blurry or the brick is a different color, you can't find it.
New Way: You have the digital file of the brick. You tell the robot, "Find anything that looks like this shape." The robot scans the pile, ignores the colors, and grabs the correct brick instantly, even if it's buried under other toys.

The Results

The researchers tested this on:

Custom 3D printed parts: Where every single item might look slightly different.
Industrial metal parts: Where parts are shiny, metallic, and often look very similar to each other.

In these tests, the "Blueprint Robot" was much better at finding the right parts than robots that relied on words or photos. It didn't get confused by color changes or messy backgrounds.

The Bottom Line

This paper teaches robots to stop judging books by their covers. Instead of looking at the color or texture of an object, they now look at the mathematical shape defined by the engineer's original design. This makes them perfect for factories where parts change colors, materials, or finishes, but their shape stays the same.

1. Problem Statement

The paper addresses a critical limitation in robotic perception for industrial and manufacturing environments (e.g., rapid prototyping, small-batch production): the inability of current promptable segmentation models to handle objects defined by geometry rather than appearance.

Limitations of Language Prompts: Natural language is often insufficient for describing engineered components, which may lack specific names or are distinguished by subtle geometric differences that vision-language models struggle to resolve.
Limitations of Image Exemplars: Standard "exemplar" prompting (using a reference image) relies on appearance cues like color and texture. In industrial settings, the same CAD part is often produced in different materials, colors, or surface finishes. Consequently, appearance-based methods fail when the query object's visual properties differ from the reference exemplar.
The Gap: While objects are canonically defined by precise CAD models (geometry), existing segmentation frameworks do not utilize these models as direct prompts for instance segmentation.

2. Methodology: CAD-Prompted SAM3

The authors propose a framework built upon SAM3 (Segment Anything Model 3) that replaces text or image exemplars with canonical multi-view renderings of a CAD model as the prompt input.

A. Architecture Overview

The system retains the core components of SAM3 (Image Encoder, Fusion Transformer, Detection Head, Mask Decoder) but introduces a new Geometry-Conditioned Prompt Construction pipeline:

Multi-View Rendering: Given a target 3D mesh $M$ , the system renders it from $V=12$ canonical viewpoints using Blender. This captures diverse geometric perspectives.
View Encoding: Each rendered view is processed by the SAM3 Image Encoder. To emphasize geometry, 25 point prompts are sampled within the foreground mask of each view. These are passed through the GeometryEncoder to produce geometry-aware embeddings ( $E_v$ ).
Cross-Image Fusion: Unlike standard SAM3 which fuses exemplars from the same image, this method treats the CAD-derived embeddings as "concept tokens." They are injected into the Fusion Transformer alongside the query image tokens ( $T_I$ ). The transformer uses cross-attention to align the geometric priors with the visual features of the query image.
Single-Stage Prediction: The fused features are processed by the detection head and mask decoder in a single forward pass to generate instance masks.

B. Training Strategy

The model is fine-tuned using a large-scale synthetic dataset to bridge the domain gap between CAD renderings and real-world images.

Data Generation: 9,000 CAD models from the ABC Dataset are used. Scenes are generated in Isaac Sim with Automated Domain Randomization (ADR) across 21 parameters (lighting, PBR materials, distractors, camera intrinsics).
Appearance Randomization: Crucially, object colors and textures are randomized independently of the mesh identity. This forces the model to rely on geometry rather than color/texture cues.
Two-Stage Optimization:
- Stage 1 (Score-Weighted Mask Supervision): Uses a score-weighted loss to stabilize early training, encouraging the model to learn basic geometric alignment without strict instance assignment.
- Stage 2 (One-to-Many Matching): Switches to a strategy allowing up to $K=5$ predictions per ground-truth instance. This prevents prediction collapse (where multiple predictions map to a single object) and enables robust multi-instance detection.

3. Key Contributions

CAD-Prompted Segmentation Formulation: The first framework to treat canonical multi-view CAD renderings as a prompt modality, enabling segmentation conditioned purely on geometric structure.
Geometry-Aware Prompting Mechanism: A novel pipeline that encodes CAD meshes into embeddings and fuses them with query images via a transformer, effectively decoupling shape from surface appearance.
Synthetic Training Pipeline: A robust data generation strategy using domain randomization to train models that generalize across varying materials and finishes, solving the "appearance variation" problem in manufacturing.
Single-Stage Performance: Demonstrates that geometry-conditioned segmentation can be achieved in a single forward pass, maintaining the efficiency of foundation models.

4. Experimental Results

The method was evaluated on a custom 3D printing dataset and two standard industrial benchmarks (T-LESS and ITODD).

Custom 3D Printing Dataset:
- Setup: 8 unseen objects, printed in different colors, captured in cluttered scenes.
- Results: CAD-Prompted SAM3 achieved a Panoptic Quality (PQ) of 0.7385 and F1 of 0.7636.
- Comparison: Significantly outperformed Matcher (PQ 0.6057), PerSAM (PQ 0.2846), and direct SAM3 image exemplar prompting (PQ 0.1713). The drop in performance for image-exemplar methods highlights their failure when appearance changes.
Industrial Benchmarks (T-LESS & ITODD):
- T-LESS (Texture-less, rigid objects): Achieved the highest PQ (0.2799) and F1 (0.3329) among all baselines.
- ITODD (Metallic, grayscale, cluttered): Achieved PQ 0.4921 and F1 0.6090, showing substantial improvement over baselines.
- Significance: The consistent gains on texture-less and metallic objects confirm that the model successfully leverages geometric priors where appearance-based cues fail.

5. Significance and Conclusion

This work fundamentally shifts how industrial objects are perceived in robotic systems. By aligning the segmentation prompt with the canonical definition of the object (the CAD model) rather than its transient visual appearance, the framework solves the "open-set" problem for manufacturing.

Practical Impact: It enables reliable robotic manipulation in rapid prototyping and small-batch manufacturing where collecting annotated data for every new color or material variation is impractical.
Future Directions: The authors suggest moving beyond rasterized renderings to directly encode meshes or point clouds into the prompt space, further tightening the integration between geometric reasoning and perception.

In summary, CAD-Prompted SAM3 proves that geometry is a more robust and expressive prompt modality than language or appearance for industrial applications, offering a scalable solution for next-generation robotic perception.