🤖 machine learning

Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data

This paper introduces Trinity, a transformer-based architecture that unifies class-specific semantic segmentation and class-agnostic terrain segmentation using a new synthetic dataset (RUGDSynth) and a real-world annotated dataset (EXTerra) to enable robot-agnostic terrain understanding for unstructured outdoor environments.

Original authors: Marcus G Müller, Wout Boerdijk, Maximilian Durner, Riccardo Giubilato, Abel Gawel, Wolfgang Stürzl, Roland Siegwart, Rudolph Triebel

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Marcus G Müller, Wout Boerdijk, Maximilian Durner, Riccardo Giubilato, Abel Gawel, Wolfgang Stürzl, Roland Siegwart, Rudolph Triebel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to navigate a wild, messy forest. To get from point A to point B, it needs to understand the ground in two very different ways at the same time.

The Problem: The "One-Size-Fits-None" Dilemma
Current robot vision systems are like rigid rulebooks.

The "Labeler" approach: Some systems only look for specific, named things like "tree," "rock," or "fence." But in the wild, the ground isn't just a list of items; it's a mix of mud, sand, and grass that blends together. If the robot sees a patch of mud that looks like sand, a rigid labeler might get confused because it doesn't fit a pre-written category.
The "Traversability" approach: Other systems try to decide if the ground is "walkable" or "not walkable." But this is like asking a question that only makes sense for one specific person. A dirt path is easy for a truck but impossible for a bicycle. If you train a robot to know what's walkable for its specific wheels, that knowledge doesn't transfer to a different robot with different legs or wheels.

The result? Every time you change the robot or the environment, you have to start over, collecting new data and re-teaching the robot from scratch.

The Solution: "Trinity"
The authors propose a new system called Trinity. Think of Trinity as a robot's brain that wears two pairs of glasses simultaneously:

The "Specific" Glasses (Class-Specific): These glasses look for named objects that matter for the mission, like "rocks," "trees," or "fences." It's like a librarian who knows exactly where the "Fiction" and "History" sections are.
The "General" Glasses (Class-Agnostic): These glasses ignore names and just look at texture and color. They group the ground into visual chunks based on how it looks, not what it's called. It's like a painter who sees a patch of "brown rough stuff" and a patch of "green soft stuff" without needing to know if they are technically "mud" or "grass." This view is universal; it works for a truck, a drone, or a rover because it's just describing what the eye sees.

The Secret Sauce: Synthetic Training
Teaching a robot to do this in the real world is a nightmare. You'd have to send humans into muddy fields to draw thousands of lines on photos, labeling every patch of ground. It's slow, expensive, and boring.

To solve this, the team built a virtual playground (using a simulator called OAISYS). They created a massive library of fake outdoor worlds filled with digital trees, rocks, and 200+ different types of ground textures. They call this dataset RUGDSynth.

The Analogy: Imagine training a pilot. Instead of crashing a real plane 10,000 times to learn how to land, you put them in a flight simulator. The simulator can generate infinite different weather conditions and runways instantly. Trinity learned its "ground sense" in this digital sandbox, learning to recognize visual patterns without needing a human to label every single pixel.

The Results: A New Kind of Map
The team tested Trinity on real-world data, including a planetary exploration lab (a fake Mars environment).

The Outcome: Trinity successfully split the world into "named objects" (like a fence) and "visual terrain patches" (like a rough, rocky area) in one go.
The Comparison: They compared Trinity to other top systems. While other systems struggled when the ground looked fuzzy or when boundaries were unclear, Trinity kept its cool, correctly identifying the visual texture of the ground even when it didn't know the specific name for it.

In a Nutshell
Trinity is a robot vision system that stops trying to force the messy, natural world into a rigid list of categories. Instead, it learns to see the world in two layers: the objects that are important (like trees) and the visual texture of the ground (like rough or smooth). By training on a massive, computer-generated world first, it learns to understand terrain so well that it can be used by different robots in different places without needing to be re-taught every time.

Technical Summary: Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments

Problem Statement
Mobile robots operating in unstructured outdoor environments require robust terrain understanding for navigation, state estimation, and mission planning. Existing vision-based approaches face two primary limitations:

Traversability Estimation: Current methods often rely on robot-specific annotations or semantic class mappings. This creates a dependency where models must be retrained or re-annotated when robot capabilities change or when deployed in new environments, limiting transferability and scalability.
Semantic Segmentation: Standard methods focus on predefined semantic classes (e.g., "road," "building"). However, outdoor terrains often exhibit irregular regions, gradual transitions, and visual similarities that make fixed taxonomies insufficient. Furthermore, the traversability of a terrain (e.g., dirt or grass) is context-dependent, varying with environmental conditions and the specific robot platform, making absolute semantic labels difficult to define universally.

Methodology: Trinity-Net
The authors propose Trinity-Net, a unified transformer-based architecture designed to jointly perform class-specific (CS) semantic segmentation and class-agnostic (CA) terrain segmentation without relying on a fixed terrain taxonomy.

Architecture: The network utilizes a frozen DINOv2 image encoder to extract features. These features are processed by a Split Transformer that implicitly divides the feature space into CS and CA components using learnable query sets ( $q_{CS}$ and $q_{CA}$ ).
- Split Mechanism: An auxiliary loss is applied to the updated query embeddings to encourage the separation of CS and CA regions. The queries are then expanded and passed to two subsequent modules: the CS-Transformer (handling mission-critical, platform-independent categories like trees or rocks) and the CA-Transformer (handling visually coherent ground regions without predefined labels).
- Query Aggregation: To handle the variability of real-world terrains, multiple queries are used per region. The outputs are aggregated, upsampled, and concatenated to form a unified prediction tensor.
- Matching: Since the CA component does not enforce a fixed alignment between queries and regions, a Hungarian matcher is used to align predicted CA regions with ground-truth masks before applying cross-entropy loss.
Data Generation (RUGDSynth): To train the CA component, the authors extended the OAISYS simulator. They introduced RUGDSynth, a large-scale synthetic dataset containing over 42,000 samples across 4,500 worlds.
- Material-Based Labeling: Unlike typical object-based simulators, OAISYS uses material-based semantics, allowing for the simulation of diverse ground textures (e.g., sand, mud, gravel) essential for class-agnostic learning.
- Enhancements: The simulator was extended with a physics-based placement module (MeshPhysicsScatter) for realistic object spawning and a custom material module (MaterialCSCATerrain) to randomize ground textures and scales, ensuring high visual diversity.
Evaluation Metric: Recognizing that standard Intersection over Union (IoU) is insufficient for class-agnostic tasks where the number of predicted regions is arbitrary, the authors introduce Residual Recall (resR). This metric aggregates all unmatched prediction masks to quantify the surplus of predicted regions compared to ground truth.

Key Contributions

Trinity-Net: A unified architecture that decouples robot-agnostic terrain representation from mission-relevant semantic prediction, enabling the learning of visual terrain priors that can be combined with robot-specific experience.
RUGDSynth: A synthetic dataset derived from the extended OAISYS simulator, providing class-agnostic terrain samples and class-specific categories to facilitate large-scale training.
EXTerra Dataset: A real-world dataset collected from a planetary exploration analog mission site, annotated with both class-specific and class-agnostic labels, used to evaluate domain shift capabilities.

Results
Experiments were conducted on the RUGD benchmark and the EXTerra dataset.

Class-Specific Performance: On the RUGD dataset, Trinity-Net achieved the highest mean IoU (51.80%) among compared models (m1, CRLNet, and SAM), demonstrating strong performance in identifying specific objects like fences, buildings, and vehicles.
Class-Agnostic Performance: Trinity-Net significantly outperformed the Segment Anything Model (SAM) and other baselines in class-agnostic segmentation, achieving a mean IoU of 69.01% on RUGD. Qualitative results indicate that while SAM tends to oversegment foreground details or struggle with fuzzy terrain boundaries, Trinity-Net effectively distinguishes terrain regions.
Synthetic Data Relevance: Training exclusively on RUGDSynth yielded better metrics than training solely on real RUGD data, indicating the high value of the synthetic data. The best performance was achieved by combining both datasets, suggesting they complement each other.
Domain Transfer: The model trained on synthetic and RUGD data successfully transferred to the EXTerra dataset (planetary exploration context), outperforming SAM in class-agnostic tasks even when boundaries were visually ambiguous.

Significance and Claims
The paper claims that Trinity-Net offers a flexible approach to scene understanding that avoids the rigidity of fixed terrain taxonomies. By learning robot-agnostic visual priors, the system can serve as a foundation for downstream tasks such as traversability estimation, visual odometry, and mission planning, adaptable to different robot platforms and environments. The authors posit that the proposed task formulation, along with the release of the RUGDSynth and EXTerra datasets, will stimulate further research in unstructured outdoor environments and contribute to advancing the autonomy of future robotic systems. The work emphasizes that while semantic labels and traversability are context-dependent, visual appearance-based segmentation provides a stable, transferable representation.

More like this