💻 computer science

Technical Report for ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge: Exploring Query-Based Segmentation and Increased Spatial Context for Outdoor Scene Understanding

This technical report details a submission to the ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge that achieves a 69.6% mIoU by enhancing a SegFormer baseline with larger training crops, a transition to the query-based Mask2Former architecture, and test-time augmentation to improve outdoor scene understanding.

Original authors: David Pascual-Hernández, Roberto Calvo-Palomino, Inmaculada Mora-Jiménez, Jose María Cañas-Plaza

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: David Pascual-Hernández, Roberto Calvo-Palomino, Inmaculada Mora-Jiménez, Jose María Cañas-Plaza

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to "see" the world outside, not just as a blur of green and gray, but as a detailed map of specific things: a specific type of bush, a wet patch of mud, a distant tree, or a specific kind of road sign. This is the challenge the authors of this paper tackled for a robotics competition called GOOSE.

Here is a simple breakdown of what they did, using everyday analogies.

The Problem: The "Blurry Vision" Challenge

Outdoor environments are messy. Unlike a city street with clear lines, nature is full of things that look very similar. Is that a pile of dirt or a rock? Is that a wet spot on the road or a puddle? Is that a tree in the foreground or part of the distant forest?

The authors needed to build a computer vision system that could tell these subtle differences apart. They had a huge library of photos (over 13,000) taken by robots in different weather and terrains, but the task was still incredibly hard because the "classes" (categories) were so specific and numerous.

The Solution: A Three-Step Upgrade

The team started with a standard, reliable model (like a good pair of glasses) and upgraded it in three specific ways to give it "super-vision."

1. Changing the Brain: From "Pixel-by-Pixel" to "Query-Based"

The Old Way (SegFormer): Imagine trying to identify a person in a crowd by looking at every single pixel of their shirt, one by one, and guessing what it is. This is slow and often leads to mistakes because you lose the big picture.
The New Way (Mask2Former): The authors switched to a "query-based" system. Think of this like a detective who asks specific questions: "Where is the car?" "Where is the tree?"
Instead of guessing every pixel, the model sends out "detectives" (queries) that look for whole objects. They focus on the shape and the whole area of the object. This helped the robot understand that a "pole" is a single, thin object, rather than just a collection of gray pixels.

2. Zooming Out: The "Wide-Angle Lens"

The Issue: If you take a photo of a tree trunk but zoom in so close that you only see the bark, you might think it's a rock or a wall because you can't see the leaves or the sky.
The Fix: The authors stopped training the model on tiny, zoomed-in squares of the image. Instead, they fed it much larger chunks of the photo (like switching from a 512x512 pixel square to a 1024x1024 one).
The Result: This gave the model "context." It could see that the "rock" was actually a tree trunk because it could also see the branches and the sky above it. It's the difference between trying to identify a person by just their shoe versus seeing their whole body and where they are standing.

3. The "Second Opinion" Trick (Test-Time Augmentation)

The Trick: Before the model makes its final guess, the authors showed it the same image three times: once normally, once slightly zoomed out, and once flipped sideways.
The Result: It's like asking three different people to look at a blurry photo and then taking a vote on what they see. If all three agree, the answer is likely correct. This made the final result much more reliable.

The Results: How Well Did It Work?

The team tested their new system against the old one:

The Upgrade: Simply zooming out (using larger images) improved the score significantly. Switching to the "detective" style (Mask2Former) improved it even more.
The Final Score: Their final system achieved a score of 69.6% on the official test. In the competition, this landed them in 5th place out of all the teams.
What it got right: It was excellent at identifying big, common things like the sky, vegetation, and terrain (getting over 90% accuracy).
What was still hard: It struggled with rare or tricky things, like "animals" or "water," which is expected because those are hard to spot and appear less often in the training photos.

The Takeaway

The paper proves two main things for robots navigating the outdoors:

Context is King: You can't understand a small part of nature if you can't see the whole scene around it. Giving the robot a wider view makes it much smarter.
Ask the Right Questions: Instead of just scanning pixels, using a system that actively looks for specific objects (queries) creates much cleaner and more accurate maps of the world.

The authors have shared their code and "brain weights" online so other researchers can use this "super-vision" to help their own robots navigate the wild.

Technical Summary: GOOSE 2D Fine-Grained Semantic Segmentation Challenge Submission

Problem Statement

The paper addresses the GOOSE 2D Fine-Grained Semantic Segmentation Challenge, organized for the ICRA 2026 Workshop on Field Robotics. The core problem involves semantic segmentation in unstructured outdoor environments, which presents unique difficulties compared to urban settings. These challenges include:

Fine-grained discrimination: Distinguishing between a large number of visually similar terrain and obstacle classes (56 fine-grained classes and 11 broader categories).
Long-tailed distributions: A significant imbalance where some classes are abundant while others are severely underrepresented.
Cross-domain robustness: The need for generalization across diverse environments, weather conditions, and robotic platforms.
Data scarcity: Limited availability of annotated data for outdoor scenes has historically hindered progress.

The challenge utilizes the unified GOOSE and GOOSE-Ex datasets, comprising over 13,000 densely annotated images captured from four distinct camera setups.

Methodology

The authors' approach evolved from a standard baseline to a more sophisticated architecture, focusing on two primary variables: the model architecture and the spatial context provided during training.

1. Architectural Transition: SegFormer to Mask2Former

Baseline (SegFormer): The authors initially utilized SegFormer (B2 variant, pretrained on Cityscapes), a hierarchical Transformer encoder with a lightweight MLP decoder. This served as a computationally efficient baseline for rapid exploration.
Proposed Model (Mask2Former): The primary architectural shift was to Mask2Former (using a Swin-L backbone, pretrained on Cityscapes). This model treats segmentation as a query-based mask classification problem. Instead of pixel-level classification, it uses a fixed set of queries to predict segmentation masks and semantic categories via masked attention. The authors hypothesize this is superior for fine-grained tasks because it reasons over coherent semantic regions, leveraging object extent, shape, and context to disambiguate visually similar classes.

2. Training Configuration and Spatial Context

Crop Size Analysis: A critical component of the study was the empirical analysis of training crop sizes. Models were trained with crops of either 512 × 512 or 1024 × 1024 pixels. The authors posited that larger crops preserve scene-level contextual cues necessary to disambiguate visually similar categories.
Data Augmentation: A robust augmentation pipeline was employed, including random rescaling, horizontal flipping, color jittering, and image quality perturbations (Gaussian noise, blur, sharpness).
Test-Time Augmentation (TTA): The final submission utilized TTA with three scaling factors (0.75, 1.0, 1.25) and horizontal flipping to enhance prediction robustness.
Optimization: Models were trained using AdamW with a cosine learning rate decay over 50 epochs. SegFormer used pixel-wise cross-entropy loss, while Mask2Former utilized a weighted combination of cross-entropy, binary cross-entropy mask loss, and Dice loss.

Key Results

The authors conducted an ablation study on the validation set (1,369 images) and reported final results on the challenge test set.

Quantitative Performance

Impact of Crop Size: Increasing the training crop size from 512×512 to 1024×1024 yielded significant gains. For SegFormer, the composite mIoU ( $mIoU_{comp}$ ) increased from 58.4% to 61.5%. For Mask2Former, this increase was even more pronounced, jumping from 67.5% to 73.4%.
Architecture Comparison: Transitioning from SegFormer to Mask2Former (at 512×512) resulted in a +9.1 point increase in $mIoU_{comp}$ .
Final Submission: The combination of Mask2Former, large-crop training, and TTA achieved a final $mIoU_{comp}$ of 69.6% on the test set, securing 5th place in the competition.
- Coarse mIoU: 76.6%
- Fine mIoU: 62.7%

Per-Category Observations

High Performance: Frequent categories like Terrain, Vegetation, and Sky achieved over 90% IoU.
Challenges: Underrepresented classes such as "Animal" (26.9%) and "Water" (58.8%) remained difficult.
Semantic Ambiguity: A notable gap existed between "Vehicle" (92.6% IoU) and "Object" (62.7% IoU). The authors attribute the lower performance on "Object" to its heterogeneous nature, which creates ambiguous boundaries with other categories.

Qualitative Findings

Structural Preservation: Mask2Former demonstrated superior ability to preserve thin structures (e.g., poles, wires) compared to SegFormer, attributed to its pixel decoder which avoids losing high-resolution features.
Coherence: Mask2Former produced more spatially coherent masks, correctly segmenting complex shapes (e.g., trailers) where SegFormer produced fragmented results.
Contextual Reasoning: Large-crop training enabled the model to distinguish between visually similar local appearances based on global context. For example, the model could differentiate between "foreground trees" and "forest" (where individual trees are indistinct) by analyzing the visibility of trunks and distance to the vehicle. Similarly, it correctly identified a blurry wall as a wall rather than a hedge by leveraging the presence of the road and the structure's overall shape.

Significance and Claims

The paper claims that its findings provide a strong baseline for fine-grained semantic segmentation in outdoor environments. The significance of the work is summarized as follows:

Effectiveness of Query-Based Approaches: The study demonstrates that query-based mask classification (Mask2Former) significantly outperforms conventional dense prediction architectures (SegFormer) for fine-grained outdoor scene understanding.
Critical Role of Spatial Context: The authors highlight that increasing the spatial context during training (via larger crop sizes) is essential for fine-grained disambiguation. This allows models to leverage scene-level cues to separate visually similar elements that belong to different semantic categories.
Reproducibility: To facilitate future research, the authors have made their code and weights publicly available, establishing a reproducible foundation for further work in this domain.

The paper concludes that while challenges remain for long-tailed and ambiguous classes, the combination of query-based architectures and expanded spatial context offers a robust path forward for outdoor scene understanding.