💻 computer science

SAM3 Self-Distillation for Fine-Grained GOOSE 2D Semantic Segmentation

This paper presents the 4th-place ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation solution, which achieves a 69.73% mIoU by adapting the SAM3 foundation model with a lightweight decoder and enhancing performance through a self-distillation scheme, image-level multi-scale test-time augmentation, and aggressive photometric distortion.

Original authors: Xuesong Wang

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xuesong Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to drive off-road, but instead of just seeing "a tree" or "a rock," the robot needs to distinguish between 64 specific types of things, like "low grass," "bushes," "moss," and "kick-scooters." This is the challenge the author, Xuesong Wang, tackled for the ICRA 2026 GOOSE competition. The goal was to build a system that can label every single pixel in a camera image with the correct object name.

The author's entry took 4th place with a score of 69.73%. Here is how they did it, explained through simple analogies.

The Core Idea: A Smart Student and a Super-Teacher

The team didn't build their robot's brain from scratch. Instead, they used a pre-trained "super-teacher" called SAM3 (Segment Anything Model 3). Think of SAM3 as a genius artist who has seen millions of images and knows exactly how to draw the outline of almost any object if you point to it.

However, this genius artist is a bit rigid; it only works well on specific types of prompts and doesn't know the specific 64 "off-road" classes the robot needs to learn.

The Solution:

The Student: They took the "brain" (the image encoder) of the genius teacher (SAM3) and gave it a small, simple "head" (a decoder) to learn the specific off-road classes.
Partial Training: They didn't retrain the whole genius brain. They only tweaked the top layers (the part that handles complex details) while leaving the bottom layers (which know basic shapes and edges) frozen. This is like hiring a master chef to teach a new restaurant; you don't retrain them on how to chop onions (they already know that), you just teach them your specific secret sauce.

The Three Secret Ingredients

The paper highlights three specific tricks that boosted the score:

1. The "Oracle Box" Self-Distillation (The Teacher Helps the Student)

Usually, a student learns from a teacher by watching them solve a problem. Here, the student (the robot) and the teacher (SAM3) are actually the same model family.

The Trick: Before training, the team asked the "genius teacher" (SAM3) to draw perfect outlines around objects using the correct ground-truth boxes (like a cheat sheet).
The Catch: The teacher isn't perfect at everything. It's great at drawing a "truck" or a "tree," but terrible at "fences" or "moss" (it spills outside the lines).
The Fix: The team only let the teacher help with the classes where it was clearly better than the student. For those 22 specific classes (like trucks, buses, and traffic cones), the student was forced to copy the teacher's perfect drawing. For the other classes, the student learned on its own.

2. The "Aggressive Color Makeover" (The Hardest Lesson)

Off-road scenes change drastically: snow in winter, mud in spring, bright sun in summer, and rain in autumn. A robot that relies on "green means grass" will fail when the grass is covered in snow or looks brown in winter.

The Trick: During training, the team took every image and aggressively messed with its colors. They randomly changed the brightness, contrast, saturation, and hue.
The Analogy: Imagine training a student to recognize a "stop sign" not just by its red color, but by its octagon shape. You show them the sign in black and white, in neon green, in sepia, and in the dark.
The Result: This was the single biggest improvement (+0.86 points). It forced the robot to stop guessing based on color and start recognizing shapes and textures.

3. The "Zoom-In, Zoom-Out" Trick (Multi-Scale Testing)

Most modern AI models are like cameras with a fixed lens; they can only look at an image at one specific size. If you feed them a tiny image, they get blurry; if you feed them a huge one, they get pixelated.

The Problem: The standard way to fix this is to resize the model to look at different sizes, but this model was too rigid to change its lens.
The Workaround: Instead of changing the model, they changed the image. Before feeding the image to the model, they shrank it down to 75% size and blew it up to 125% size.
The Process:
1. Take the original photo.
2. Shrink it, run it through the robot's brain, and record the answer.
3. Blow it up, run it through the brain, and record the answer.
4. Run the original size, too.
5. Average the results.
Why it works: Looking at a "traffic cone" from far away (shrunken) helps the robot see the whole object. Looking at it close up (blown up) helps it see the fine details. Combining both views makes the guess much more accurate. This added another +0.34 points without needing any extra training.

What Didn't Work?

The author was honest about what they tried that failed:

Heavier Brains: They tried using a more complex "head" (decoder) for the model, but it actually made the robot worse at spotting small, rare objects.
Other Teachers: They tested other famous AI models (like DINOv2), but none were as good as SAM3 for this specific job.
Random Resizing: Randomly resizing images during training didn't help as much as the aggressive color changes.

The Bottom Line

The robot got 4th place by using a powerful pre-trained AI as a base, letting a "genius teacher" help it learn the easy classes, forcing it to ignore color cues so it learns shapes, and using a clever "zoom trick" during the final test to see objects from multiple distances.

The only thing it still struggles with? "Moss." The robot keeps confusing moss with low grass or forest floor, likely because moss is so rare and looks so similar to other things in the dataset.

Technical Summary: SAM3 Self-Distillation for Fine-Grained GOOSE 2D Semantic Segmentation

Problem Statement

The paper addresses the ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge, which requires labeling every pixel in off-road images with one of 64 fine-grained classes (e.g., specific vegetation types, vehicle models) that aggregate into 11 coarse super-categories. The challenge is evaluated on a held-out test set of 1,815 images captured across four distinct robotic camera platforms (MuCAR-3, ALICE, and two Spot configurations).

The authors identify three primary difficulties in this domain:

Long-tailed Class Distribution: A few classes (e.g., forest, low-grass) dominate the training data, while others (e.g., kick-scooter, pipe) appear in only a few parts per million.
Visual Near-Equivalence: Many classes within the same super-category (e.g., different vegetation types) differ only subtly in canopy density or hardiness, leading to high confusion rates.
Platform Heterogeneity: Variations in resolution, exposure, and viewpoint across the four camera platforms make it difficult to create a model that generalizes without overfitting to a specific geometry.

Methodology

The proposed solution adapts the Segment Anything Model 3 (SAM3) visual foundation model for off-road segmentation. The architecture and training pipeline consist of the following components:

1. Backbone and Decoder

Backbone: The authors utilize the pretrained image encoder of SAM3. Instead of freezing it entirely or retraining from scratch, they employ partial fine-tuning: the lower layers (capturing generic visual features) remain frozen, while the upper 12 transformer blocks are fine-tuned. This balances retaining general knowledge with adapting to the off-road domain.
Decoder: A lightweight Feature Pyramid Network (FPN) style decoder is attached to the encoder. It fuses multi-resolution features into a single per-pixel class map. The authors found that a more complex decoder (UPerNet) performed slightly worse, likely due to blurring small or rare objects.

2. SAM3 Oracle-Box Self-Distillation

A novel self-distillation scheme is introduced where SAM3 itself acts as a teacher for the student model:

Mechanism: SAM3 is prompted with ground-truth bounding boxes ("oracle-box prompting") to generate high-quality soft masks for specific objects.
Selection Criteria: SAM3 is not used as a teacher for all classes. The authors pre-evaluate classes and only apply distillation for 22 classes where SAM3's oracle-box masks are demonstrably superior to the student's initial predictions (typically compact, well-defined objects like trucks, traffic signs, and trees).
Loss Function: The training objective combines standard cross-entropy loss ( $L_{CE}$ ) for all 56 classes with a distillation loss ( $L_{teach}$ ) for the selected 22 classes, weighted by $\lambda = 0.3$ .

3. Aggressive Photometric Distortion

The pipeline adopts an aggressive color augmentation strategy from a winning 2025 entry. Unlike standard methods that apply brightness, contrast, saturation, and hue perturbations simultaneously, this approach applies each perturbation independently with a probability of 0.5. This creates a richer distribution of training images, forcing the model to rely on shape and texture rather than color cues, which is critical given the varying lighting and seasonal conditions in off-road scenes.

4. Image-Level Multi-Scale Test-Time Augmentation (TTA)

Standard multi-scale TTA involves feeding rescaled images into a model. However, SAM3's encoder accepts only a fixed input size (1008×1008). The authors propose a workaround:

Process: Instead of rescaling the model input, the entire test image is rescaled to factors of {0.75, 1.0, 1.25}.
Inference: Each rescaled image is processed via sliding-window inference (tiling) at the model's native resolution.
Fusion: The resulting score maps are resized back to the original image dimensions, weighted by a Hann window to smooth tile edges, and accumulated across scales. The final prediction is the class with the highest accumulated score. This recovers the benefits of multi-scale inference without retraining the fixed-input model.

Key Results

The final model achieved a composite mean Intersection-over-Union (mIoU) of 69.73% on the official test set, securing 4th place in the challenge. The breakdown of performance gains is as follows:

Base Model (Encoder + Decoder): 67.71% composite mIoU.
+ SAM3 Self-Distillation: Improved to 68.38% (+0.67).
+ Flip Augmentation: +0.13 to +0.15 improvement.
+ Aggressive Color Augmentation: The single largest source of improvement, adding +0.86 (reaching 69.24%). This was particularly effective for rare classes (e.g., kick-scooter IoU increased from 7.90% to 29.65%).
+ Image-Level Multi-Scale TTA: Added a further +0.34 (reaching 69.73%), primarily benefiting medium-rare, well-defined objects like traffic cones and signs.

Per-Class Observations:

Successes: The model performed strongly on Sky (97.47%), Vegetation (94.01%), and Vehicle (90.94%) super-categories. Color augmentation significantly rescued rare classes.
Failures: The model struggled with moss (dropping to near 0% IoU with color augmentation) and tree-root, likely due to visual similarity with other vegetation classes and extreme rarity. Water and Animal categories also remained weak (<50% IoU).

Significance and Contributions

The paper claims three primary contributions:

Self-Distillation from the Foundation Model: Demonstrating that a foundation model (SAM3), when prompted with ground-truth boxes, can serve as a high-quality teacher for specific classes in a specialized domain, refining the student model's predictions without requiring external teacher networks.
Image-Level Multi-Scale TTA: Providing a generalizable method to enable multi-scale inference for any model locked to a fixed input size by rescaling the image rather than the model input, recovering accuracy gains without additional training cost.
Empirical Finding on Color Augmentation: Highlighting that aggressive, independent photometric distortion—previously successful in a 2025 challenge—is the single most impactful component for this specific pipeline, significantly outperforming other architectural tweaks or loss functions.

The authors note that while the approach is effective, it does not solve the fundamental issue of within-vegetation confusion for extremely rare classes (like moss), suggesting that future improvements may require additional pretraining on related off-road datasets.