💻 computer science

Instance-Guided Unsupervised Domain Adaptation for Robotic Semantic Segmentation

This paper proposes an instance-guided unsupervised domain adaptation method for robotic semantic segmentation that leverages volumetric 3D maps and zero-shot foundation models to generate consistent pseudo-labels, thereby improving adaptation performance in target environments without requiring ground-truth annotations.

Original authors: Michele Antonazzi, Lorenzo Signorelli, Matteo Luperto, Nicola Basilico

Published 2026-02-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Michele Antonazzi, Lorenzo Signorelli, Matteo Luperto, Nicola Basilico

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to recognize objects in a room, like a chair, a table, or a door. You train the robot in a "perfect" classroom (the Source Domain) where the lighting is great, the furniture is new, and everything is clearly labeled. The robot becomes a genius at this specific classroom.

But then, you send the robot out into the real world (the Target Domain). Suddenly, the lighting is dim, the furniture is old and dusty, and the robot gets confused. This is called Domain Shift. The robot's brain is stuck in "classroom mode" and can't handle the messy reality.

Usually, to fix this, you'd have to hire a human to walk around the new room, point at every object, and say, "That's a chair, that's a table." This is expensive, slow, and boring.

This paper proposes a clever way to teach the robot without hiring a human. It's like giving the robot a self-study guide that gets better the more it reads. Here is how their method works, broken down into simple steps:

1. The "3D Group Hug" (Multi-View Consistency)

First, the robot drives around the new room, taking pictures from many different angles.

The Old Way: If the robot looks at a chair from the left, it might think it's a "wall" because it's in shadow. If it looks from the right, it might think it's a "table."
The Paper's First Step: The robot builds a 3D map of the room. It takes all those different guesses from different angles and "hugs" them together. If 9 out of 10 angles say "chair," the robot decides, "Okay, it's probably a chair."
The Problem: This is good, but not perfect. Sometimes, the 3D map gets messy. It might look like a chair is half-chair and half-wall because the robot only saw the edge of it. Or, the computer rendering might create weird visual glitches (like a chair leg floating in mid-air).

2. The "Magic Detective" (Instance Refinement)

This is the paper's secret sauce. The robot realizes that objects have identity. A chair is one single thing, not a collection of random pixels.

The Analogy: Imagine the robot's 3D map is a messy sketch where a chair looks like a pile of scattered bricks. The robot then calls in a "Magic Detective" (a powerful AI called SAM - Segment Anything Model).
How the Detective Works: The detective looks at the picture and says, "Wait, those scattered bricks actually belong to one single object: a chair." It draws a perfect outline around the whole chair, ignoring the messy pixels.
The Strategy: The paper uses two ways to ask the detective for help:
1. The Grid Strategy: The detective puts a grid of dots over the whole picture and finds every object it can see, like a net catching fish.
2. The Informed Strategy: The detective looks at the robot's messy 3D map first, finds the big blobs, and then zooms in to clean them up.

3. The "Self-Correction" (Fine-Tuning)

Once the Magic Detective has cleaned up the messy 3D map and drawn perfect outlines around the objects, the robot uses these cleaned-up labels as a textbook.

It re-trains its brain using this new, high-quality "self-study guide."
Because the guide is now accurate (thanks to the detective), the robot learns the correct names for things in the new room, even though no human ever told it what they were.

Why is this a big deal?

No Humans Needed: The robot teaches itself using data it collects naturally while driving around.
Fixes "Glitches": Previous methods were great at grouping views but bad at understanding that an object is a single unit. This method fixes the "floating chair leg" and "half-wall" errors.
Better Performance: In tests, this method made the robot much smarter at recognizing objects in new environments compared to other robots that just tried to guess based on 3D maps alone.

In a nutshell: The paper teaches a robot to clean up its own messy notes by using a "Magic Detective" to group pixels into real objects, allowing the robot to learn from its own experiences without needing a human teacher.

1. Problem Statement

Robotic semantic segmentation networks, essential for tasks like navigation and manipulation, suffer from domain shift when deployed in new environments. Models trained on source datasets (e.g., controlled indoor scenes) often degrade in performance when faced with different visual distributions, lighting, or layouts in target environments.

While Unsupervised Domain Adaptation (UDA) aims to solve this without ground-truth labels in the target domain, existing robotic UDA methods face specific challenges:

Reliance on Multi-View Consistency: Previous approaches (e.g., [6]) aggregate 2D predictions into a 3D map to enforce spatial consistency. While this reduces frame-to-frame noise, it fails to resolve instance-level inconsistencies.
Artifacts and Misclassification: 3D rendering processes can introduce visual artifacts (e.g., from reflective surfaces or voxel discretization). More critically, if a robot observes an object partially (e.g., a cabinet corner) and misclassifies it repeatedly, the 3D aggregation may solidify this error, assigning the wrong label to the entire object instance.
Lack of Instance Coherence: Current methods treat pixels independently or by spatial location, failing to ensure that all pixels belonging to a single physical object share the same semantic class.

2. Methodology

The authors propose a novel framework that combines multi-view consistency with instance-aware refinement using foundation models. The pipeline consists of three main stages (illustrated in Fig. 2 of the paper):

A. Multi-View Consistent Pseudo-Label Generation

The process begins by aggregating the source model's per-frame predictions ( $Y^*$ ) into a 3D volumetric map (using Kimera Semantics).

The robot's sequence of RGB-D images and poses are projected into a TSDF (Truncated Signed Distance Function) volume.
Semantic probabilities are updated voxel-by-voxel based on the projected pixels.
A high-resolution mesh is extracted, and ray-tracing is performed from each camera pose to generate Multi-View Consistent (MVC) pseudo-labels ( $Y_{MC}$ ). This step removes frame-level noise but retains instance-level errors and rendering artifacts.

B. Instance-Aware Refinement

To correct the limitations of $Y_{MC}$ , the authors introduce a refinement stage leveraging the Segment Anything Model (SAM), a zero-shot foundation model for instance segmentation.

Goal: Enforce that all pixels within a single object instance share the most frequent semantic class found in the $Y_{MC}$ map.
Prompting Strategies: The paper proposes and compares two automated prompting strategies to query SAM:
1. Grid Strategy: A fixed grid of points is overlaid on the image to generate instance masks. This is effective for complex shapes and noisy labels but may over-segment objects.
2. Informed Strategy: Prompts (bounding boxes/centroids) are procedurally extracted from the connected clusters of the $Y_{MC}$ map. This is more conservative and prevents over-segmentation but may struggle with complex boundaries.
Refinement Logic: For each instance mask generated by SAM, the algorithm identifies the dominant semantic class from the corresponding region in $Y_{MC}$ $Y_{M C}$ . It then overwrites the pixel labels in the refined mask ( $Y_{IR}$ $Y_{I R}$ ) with this dominant class.
- Note: When using the Grid strategy, masks are sorted by size (descending) to prevent large instances from overwriting small, detailed ones.

C. Self-Supervised Fine-Tuning

The refined, instance-coherent pseudo-labels ( $Y_{IR}$ ) serve as supervision for fine-tuning the source network ( $f_{\theta_s}$ ) on the target domain data. This results in an adapted model ( $f_{\theta_t}$ ) with improved performance in the target environment.

3. Key Contributions

Novel Framework: A UDA approach for robotics that integrates multi-view 3D consistency with instance-level refinement, addressing the specific failure modes of previous 3D aggregation methods.
Foundation Model Integration: The first application of SAM (Segment Anything) in a robotic UDA pipeline to enforce instance coherence. The authors devise two complementary automated prompting strategies (Grid and Informed) to adapt SAM for this specific task without manual intervention.
Robustness to Artifacts: The method effectively mitigates rendering artifacts and systematic misclassifications caused by partial object observations in 3D maps.
Extensive Evaluation: Comprehensive experiments on real-world data (ScanNet) demonstrating superior performance over state-of-the-art baselines without requiring target domain ground truth.

4. Experimental Results

The method was evaluated on the ScanNet dataset (1513 sequences, 707 environments), using DeepLabV3+ as the backbone.

Pseudo-Label Quality (Table I):
- The baseline Multi-View Consistency ( $Y_{MC}$ ) improved raw predictions ( $Y^*$ ) by an average of 3.9%, but performance was unstable (degrading in some environments like Env 2 and 5).
- The proposed instance-refined labels ( $Y_{IR}$ ) improved $Y_{MC}$ by an average of 3.8% (Grid) and 4.1% (Informed). Crucially, refinement improved performance in all environments, whereas the baseline failed in several.
Adapted Model Performance (Table II & III):
- Same Sequence (Train/Test split): The refined method improved the baseline ( $Y_{MC}$ ) by 4.3% to 6.7% (mIoU) depending on the strategy.
- Different Sequences (Real-world deployment simulation): When training and testing were performed on entirely different trajectories within the same environment, the method outperformed the baseline by ~7% (5cm voxel) and ~4% (3cm voxel).
- Combined Strategy (IRGI): Combining both Grid and Informed prompts during training yielded the best results, achieving the highest or second-highest performance across almost all environments.
- Resolution Independence: Unlike the baseline, which is highly sensitive to voxel size (3cm vs. 5cm), the proposed method maintained consistent performance across resolutions, making it more practical for resource-constrained robots.
Qualitative Improvements:
- The method successfully corrected persistent errors where objects (e.g., doors, cabinets) were misclassified as walls due to partial views.
- It resolved boundary artifacts caused by voxel discretization and improved segmentation of objects with similar textures.

5. Significance

This work represents a significant step forward in robotic perception by bridging the gap between geometric consistency (3D mapping) and semantic instance coherence.

Practicality: It enables robots to adapt to new environments in real-time using only their own collected data, eliminating the need for expensive manual annotation.
Reliability: By leveraging foundation models (SAM) to enforce physical object boundaries, the system becomes more robust to the "noisy" nature of robotic exploration, where objects are often partially occluded.
Generalizability: The approach is not limited to a specific sensor setup or architecture, suggesting a scalable path for deploying autonomous systems in diverse, unstructured environments.