SAMa: Material-aware 3D Selection and Segmentation

The paper introduces SAMa, an optimization-free method that leverages a material-centric video dataset and 2D-to-3D lifting via depth projection to enable fast, multiview-consistent selection and segmentation of materials on arbitrary 3D assets.

The Gist

Imagine you have a magical 3D object, like a digital statue or a virtual room. You want to change just one thing: maybe you want to turn the wooden table into glass, or make the red car blue, without touching the rest of the scene.

In the past, doing this was like trying to paint a specific leaf on a tree while the wind was blowing, using a brush that kept painting the whole tree by accident. Artists had to manually click on every single pixel of the wood, frame by frame, which took hours and often looked messy.

Enter SAMa (Select Any Material). Think of SAMa as a "Magic Material Magnet" for 3D worlds.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Flat Screen" Trap

Most AI tools are great at looking at a single 2D photo. If you click on a wooden chair in a photo, the AI knows "that's wood." But if you move the camera to the side, the AI might get confused. It might think the shadow is a different material, or it might forget which chair you were talking about.

Trying to fix this usually requires a slow, heavy process where the computer has to "re-learn" the object every time you want to make a change. It's like asking a librarian to re-shelve the entire library every time you want to find one specific book.

2. The Solution: Borrowing from Video

The researchers behind SAMa had a clever idea: 3D objects look a lot like videos.
If you walk around a statue, the view changes smoothly, just like frames in a movie.

They took a powerful AI model called SAM2, which was originally trained to track objects in videos (like following a dog running through a park), and taught it a new trick: Material Tracking.

• Old SAM2: "That's a dog! Follow the dog!"
• New SAMa: "That's wood! Follow the wood, even if the camera moves or the lighting changes!"

They trained this AI on a special "movie dataset" where objects spun around, zoomed in, and zoomed out, so the AI learned that "wood" is wood, regardless of the angle.

3. The Magic Trick: The "3D Cloud"

Once the AI knows what "wood" looks like from every angle, how do we apply it to a 3D model?

Instead of trying to force the AI to understand complex 3D math directly, SAMa uses a clever shortcut:

1. The Click: You click on a spot of wood in one view.
2. The Snapshot: The AI quickly looks at a few different angles (like taking a few photos from different sides).
3. The Cloud: It projects those 2D photos into a 3D "similarity cloud." Imagine sprinkling millions of tiny, invisible dust particles around the object. Each particle knows, "I am close to the wood you clicked on."
4. The Search: When you move your camera to a new angle, the computer doesn't ask the AI to think again. It just asks the "dust cloud": "Hey, which particles are closest to this new view?"

This is incredibly fast. It's like having a map where every location is pre-labeled. You don't need to ask for directions; you just look at the map.

4. Why It's a Big Deal

• Speed: It takes about 2 seconds to select a material and start editing. Old methods could take 20 minutes or even hours.
• Consistency: If you paint the table blue, the AI ensures the entire table is blue, even the parts hidden behind a chair. It doesn't get confused by shadows or reflections.
• Versatility: It works on almost any type of 3D object, whether it's a smooth mesh (like a video game character), a "NeRF" (a photo-realistic 3D scene made from photos), or "3D Gaussians" (a new, super-fast way to render 3D).

Real-World Examples

• The Text-to-3D Fix: Imagine an AI generates a 3D chair, but it looks like a flat, dull painting. With SAMa, you can click the "wood" parts and instantly swap them for shiny, realistic PBR (Physically Based Rendering) materials, making the chair look like it belongs in a real room.
• The Gazebo Edit: In the paper, they showed a gazebo where they could click the wooden base and make it float in the air, or click the white paint and make it disappear, all while keeping the rest of the structure perfectly intact.

The Bottom Line

SAMa is like giving artists a universal remote control for 3D materials. Instead of manually painting every inch of a 3D world, you just point and click, and the AI instantly understands, "Oh, you want to change all the wood," and does the heavy lifting for you in the blink of an eye.

Technical Summary

1. Problem Statement

Decomposing 3D assets into material parts is a critical task for artists and downstream applications (e.g., editing, inverse rendering, text-to-3D workflows). However, current methods face significant challenges:

• Manual Labor: Existing workflows are highly manual.
• 2D Limitations: Most material understanding models operate in 2D. When extended to 3D, they lack multiview consistency, leading to flickering or inconsistent selections across different camera angles.
• Semantic vs. Material: Existing 3D segmentation methods often focus on semantic classes (e.g., "wood," "metal") rather than specific material appearances (e.g., distinguishing between two types of wood or different plastic textures).
• Computational Cost: Current approaches to lift 2D selections to 3D often require expensive per-asset optimization (e.g., feature field distillation or contrastive learning), taking minutes to hours per object.

2. Methodology: SAMa

The authors propose SAMa (Select Any Material), an efficient, optimization-free approach for selecting materials in arbitrary 3D representations (Meshes, NeRFs, 3D Gaussians). The method consists of three core components:

A. Video-Based Fine-Tuning (The Core Insight)

Instead of treating 3D views as independent images, the authors draw a parallel between video frames and 3D views.

• Model Base: They adapt SAM2 (Segment Anything Model 2), which is designed for temporally coherent object selection in videos.
• Custom Dataset: They created a novel, object-centric video dataset with dense, per-pixel, per-frame material annotations. This dataset includes synthetic scenes with multiple objects and materials, rendered with various camera trajectories (zoom, turntable, flyover).
• Training Strategy:
  • The pre-trained ViT image encoder is frozen to preserve rich visual priors.
  • The Memory Attention Module and Mask Decoder are fine-tuned.
  • Key Finding: Fine-tuning on videos (rather than static images) is crucial. It forces the model to utilize its memory module, ensuring that material selections remain consistent across frames (views) even when occlusions or lighting change.
  • Frame Duplication: To improve selection quality on the specific frame where the user clicks, the authors duplicate the click frame in the sequence. This forces the model to query its memory for the clicked frame, reducing noise.

B. 2D-to-3D Lifting via Similarity Point Cloud

Once the model is trained, it can generate a similarity map for any 2D view. To make this interactive and view-independent:

1. Sampling: The system renders a sparse set of views (keyframes) covering the object.
2. Projection: For each view, the 2D similarity map (indicating how similar a pixel is to the clicked material) is back-projected into 3D space using depth information, creating a 3D similarity point cloud.
3. Querying: For any novel view, the system performs a k-Nearest Neighbor (kNN) lookup in this point cloud to retrieve the similarity value for the 3D points visible in the new view.
4. Efficiency: This process avoids per-asset optimization. The point cloud is reconstructed in ~0.5 seconds, and querying new views takes 10–20 milliseconds.

C. Refinement

• kNN Voting: To generate a clean binary mask, the system uses a voting scheme: a 3D point is selected if more than half of its $k$ nearest neighbors in the point cloud exceed the similarity threshold.

3. Key Contributions

1. First Material-Aware 3D Selector: Introduces a method specifically for selecting materials based on appearance (texture/reflectance) rather than semantic class, supporting intra-class distinctions.
2. Video-to-3D Adaptation: Demonstrates that fine-tuning a video selection model (SAM2) on a custom material video dataset is the key to achieving high-quality, multiview-consistent 3D selection without per-asset optimization.
3. Optimization-Free Efficiency: Proposes a lightweight 2D-to-3D lifting strategy using a similarity point cloud and kNN lookup, reducing selection time from minutes/hours to ~2 seconds (including initialization) and enabling real-time visualization (<20ms).
4. Multi-Representation Support: The method is representation-agnostic, working seamlessly on Meshes, NeRFs, and 3D Gaussian Splatting (3DGS).

4. Results and Evaluation

The authors evaluated SAMa on NeRF datasets (NeRF, MIPNeRF-360), 3D Gaussians, and meshes, comparing it against baselines like Materialistic, SAM2, and MatSeg3D.

• Selection Accuracy: SAMa significantly outperforms baselines in mIoU (Mean Intersection over Union) and F1 Score. For example, on the MIPNeRF-360 dataset, SAMa achieved an mIoU of 0.60 compared to 0.31 for Materialistic and 0.51 for SAM2.
• Multiview Consistency: SAMa achieves consistency scores comparable to SAM2 (which is trained for video) and significantly better than static-image-based methods (Materialistic), which suffer from high Hamming distances across views.
• Robustness: The method is robust to different click locations within the same material and handles challenging conditions like occlusions, shadows, and specular reflections.
• Speed:
  • SAMa: ~2 seconds total (click-to-selection).
  • Baselines: Ranging from 20 minutes to 2 hours per asset due to optimization requirements.

5. Significance and Applications

SAMa bridges the gap between 2D material understanding and 3D asset manipulation. Its significance lies in:

• Enabling New Workflows: It allows artists to instantly select and edit specific materials on complex 3D assets (e.g., changing the wood texture on a specific part of a chair without affecting the metal legs).
• Text-to-3D Enhancement: It can automatically decompose text-to-3D outputs (which often have baked, diffuse-only textures) into material IDs, allowing for the replacement of these textures with high-quality PBR materials.
• Interactive Editing: The speed of the method enables interactive applications, such as:
  • NeRF Editing: Adjusting colors of selected materials while preserving shading.
  • 3DGS Editing: Modifying density or position of specific material groups.
  • Mesh Segmentation: Automatically generating material ID maps for UV unwrapping and texturing.

In conclusion, SAMa represents a paradigm shift in 3D editing by leveraging video priors to achieve fast, consistent, and material-specific selection across diverse 3D representations without the computational burden of per-asset optimization.