Technical Summary: Human-in-the-Loop Atlas-Based 3D Asset Segmentation for Interactive Content Workflows

Problem Statement

Segmenting 3D assets into meaningful regions remains a significant challenge, particularly when segmentation criteria are application-dependent and require user control. Existing 3D segmentation methods face several limitations:

• Dataset Specificity: Many approaches require retraining for new object categories.
• Geometric Assumptions: Primitive-based decompositions assume simple shapes and struggle with complex, organic, or irregular real-world surfaces.
• Lack of User Control: Recent zero-shot methods transferring 2D foundation model features to 3D backbones rely on automatic semantic labeling, offering limited support for user-defined boundaries or application-specific segment definitions.

For interactive media, game, and XR workflows, segmentation is often a controllable intermediate representation needed for material editing, stylization, or texture-space operations, rather than an end goal. While 2D foundation models (e.g., SAM 2) have advanced image segmentation, fully automatic 3D application remains insufficient due to incomplete masks, over-segmentation, and the variability of what constitutes a "meaningful" segment across different domains (e.g., cultural heritage conservation vs. game asset creation). Furthermore, existing atlas-based workflows often lack explicit viewpoint optimization for coverage and do not support interactive correction.

Methodology

The authors propose a human-in-the-loop pipeline to generate a segmented 2D parameterized atlas from a 3D model. The pipeline consists of four stages:

1. View Selection (Path-Planning-Inspired)

To ensure full surface coverage with a minimal set of viewpoints, the authors formulate the problem as a set cover problem over the model's surface, inspired by robotic arm inspection tasks.

• Process: The 3D model is enclosed in a bounding sphere, and a camera sphere is constructed at a fixed radius. The model is remeshed to generate uniform sample points.
• Visibility Mapping: Raycasting is performed from each surface point to every candidate camera position to build visibility maps.
• Greedy Selection: A greedy strategy iteratively selects the camera position that covers the maximum number of previously uncovered surface points. This continues until all sampled points are covered.
• Output: A minimal set of camera positions is used to render images, which are then used for the subsequent steps.

2. AI-Assisted Segmentation

Rendered views are segmented using an interactive workflow integrating Label Studio and the SAM 2 foundation model.

• Interaction: Users provide minimal prompts (point clicks or bounding boxes) to indicate regions of interest. SAM 2 generates initial masks.
• Refinement: Users iteratively refine predictions to correct errors (e.g., incomplete masks, incorrect boundaries) and adapt segmentation to specific domain criteria.
• Consistency: Predictions are refined across the set of views to ensure cross-view consistency before back-projection.

3. Back-Projection onto UV Atlas

The segmented color images are projected back onto the model's 2D parameterized atlas (UV map).

• Correspondence: For each pixel in a segmented image, the corresponding mesh face is identified using an index pass rendered from the same viewpoint.
• Mapping: The segment color is written to the corresponding location in UV space using the face's texture coordinates.
• Merging: Partial atlas images from all selected views are merged into a single, unified segmented atlas. The greedy view selection ensures that the union of projections provides complete coverage of the UV atlas.

4. Downstream Applications

The resulting segmented atlas enables various tasks in UV space, including:

• Segment-wise material assignment (e.g., distinct roughness, metallicity, albedo).
• Per-region style transfer.
• Texture synthesis constrained to segments.
• Automated 3D surface annotation for dataset generation.

Key Contributions

1. View Selection Algorithm: A path-planning-inspired algorithm that formulates viewpoint planning as a set cover problem, using a greedy strategy to obtain a compact set of camera positions ensuring full surface coverage.
2. End-to-End Human-in-the-Loop Pipeline: A complete workflow integrating SAM 2 and Label Studio for iterative refinement of segmentation predictions, producing a segmented 2D parameterized atlas suitable for downstream texture-space editing.
3. Demonstration-Based Evaluation: An evaluation on eight diverse cultural heritage objects characterizing annotation effort, output adequacy, and recurring failure modes across different geometries and appearance conditions.

Results and Evaluation

The pipeline was evaluated on eight cultural heritage objects (e.g., Michelangelo's David head, Nefertiti bust, Victorian chair, medieval sword) with varying geometries and appearances. The evaluation was a demonstration-based technical assessment rather than a participant-based user study.

• Performance: The pipeline successfully generated usable segmented atlases for all objects. Total annotation time ranged from approximately 15 to 35 minutes per object (mean ~21 minutes).
• Success Cases: Simpler objects with clear geometry (e.g., medieval sword) required minimal correction. Larger contiguous regions with clear appearance cues were segmented reliably by SAM 2.
• Challenges and Failure Modes:
  • Fine Structures: Small details (e.g., eyes of the David statue, wings of an angel) often required manual annotation or refinement.
  • Weak Boundaries: Objects with low texture/contrast (e.g., marble surfaces) or weak appearance boundaries posed difficulties for automatic segmentation.
  • Cavities and Occlusion: Underside cavities (e.g., Victorian chair) and narrow structures (e.g., straps on a shield) were challenging, sometimes requiring manual boundary correction due to unfavorable viewing angles or tool limitations.
  • Projection Artifacts: Localized artifacts occurred in regions with shallow viewing angles or self-occlusion.

Significance and Claims

The paper claims that this approach offers a practical solution for interactive content workflows where segmentation criteria are not static but depend on user intent and application needs. By combining a mathematically optimized view selection strategy with a human-in-the-loop annotation workflow, the method bridges the gap between the efficiency of foundation models and the necessity of human control.

The authors modestly state that while the pipeline generates usable atlases across diverse assets, it does not yet achieve fully automatic segmentation for complex cases. The primary value lies in reducing manual effort for large regions while providing a structured mechanism for expert-guided refinement of difficult areas. The work suggests that the annotated data produced could eventually be used to embed task-specific priors into foundation models, potentially reducing human intervention in the future.