💻 computer science

SUNSET - A Sensor-fUsioN based semantic SegmEnTation exemplar for ROS-based self-adaptation

The paper presents SUNSET, a ROS2-based exemplar designed to facilitate the rigorous and repeatable evaluation of architecture-based self-adaptation in robotic systems by implementing a sensor-fusion semantic segmentation pipeline capable of simulating and recovering from multiple concurrent failures through fault injection.

Original authors: Andreas Wiedholz, Rafael Paintner, Alwin Hoffmann, Carlos Hernandez, Tobias Huber

Published 2026-07-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Andreas Wiedholz, Rafael Paintner, Alwin Hoffmann, Carlos Hernandez, Tobias Huber

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a spaceship, but instead of a sleek control room, your ship is a robot that has to navigate a chaotic, ever-changing world. Sometimes the robot's eyes (cameras) get blurry, sometimes its brain (software) glitches, and sometimes the whole system gets confused by too many things happening at once. In the world of robotics, this is a huge problem. Robots used to work in perfect, predictable factories, but now they are being sent out into the messy real world—flying over cities, driving on dirt roads, and helping people. When things go wrong, it's often easy to see that the robot is failing (like a car sputtering), but it's incredibly hard to figure out why it's failing. Is the camera broken? Is the software confused? Did a bird hit the lens?

This is where the field of "self-adaptive systems" comes in. Think of it like giving the robot a built-in mechanic and a detective in its head. If the robot senses something is wrong, it tries to fix itself without waiting for a human to press a button. It might restart a broken part, switch to a backup camera, or change how it thinks. But to build a robot that can do this, engineers need a safe place to test their ideas. They need a "training ground" where they can break things on purpose to see if the robot's self-fixing skills actually work. This is exactly the challenge tackled in a new paper called SUNSET.

The Robot's "Self-Healing" Training Ground

Meet SUNSET (Sensor-fUsioN based semantic SegmEnTation). It's not a robot itself, but rather a super-smart simulation—a digital playground built for testing how well robots can fix themselves when things go haywire. The authors, a team of researchers from Germany and the Netherlands, created this "exemplar" (which is just a fancy word for a perfect example or a test model) to solve a specific headache in robotics: What happens when a robot breaks in two different ways at the exact same time, and you don't know which one is the real culprit?

The Scenario: A Drone with a Mission

Imagine a drone flying over a town, trying to understand what it sees. Is that a car? A tree? A person? To do this, the drone uses a "semantic segmentation" pipeline. In plain English, this means the drone takes pictures from its cameras and uses a trained AI brain to label every single pixel in the image. It's like the drone is playing a high-speed game of "I Spy" where it has to identify everything in the world instantly.

In the SUNSET simulation, this drone is flying through a digital city (using a dataset called SynDrone) and looking at the world through two types of eyes: a standard color camera (RGB) and a depth camera (which sees how far away things are). The drone fuses these two views together to get a perfect 3D understanding of its surroundings.

The Chaos: Breaking Things on Purpose

Here is where the fun begins. The researchers built a "Fault Injector"—a digital saboteur that can break the drone's system in 11 different ways. But the tricky part is that these breaks don't always look like the break itself.

Think of it like a car that won't start. The symptom is "the car won't start." But the cause could be a dead battery, a broken starter motor, or a flooded engine. In SUNSET, the researchers simulate scenarios where:

The symptoms are ambiguous: The drone's vision gets blurry. Is it because the camera lens is dirty? Because the software is crashing? Or because the data is getting lost in the wire? The system sees the blur, but it doesn't know the root cause.
Multiple things break at once: The drone might have a camera glitch and a software crash happening simultaneously. This is called a "concurrent failure." Most old test models only let you break one thing at a time, which is too easy. SUNSET lets you break multiple things at once, forcing the robot's "self-adaptation" brain to prioritize and figure out which fix to apply first.

The Toolkit: How the Robot Fixes Itself

The paper shows that SUNSET allows a "managing system" (the robot's internal mechanic) to try four main types of fixes:

Reparametrization: Tweaking the settings. Like turning the focus knob on a camera or adjusting the volume.
Changing Communication: Switching the channel. If the data stream is blocked, the robot can switch to a different data pipe.
Restarting Components: Turning a specific part off and on again. It's like rebooting a frozen computer program.
Redeploying: This is the nuclear option. It's like pulling the plug on a specific hardware part and restarting the whole machine. The paper notes that this is heavy-handed and takes time, so the robot should only do it if necessary.

The researchers tested a basic "baseline" robot (a simple version of the self-adaptation system) to see how it handled these disasters. They ran 54 different scenarios, mixing and matching the faults to create every possible combination of chaos.

The Results: It's Hard, But It Works

The results, presented in the paper, show that while the robot can fix itself, it's not perfect.

When the drone flew without any problems, its vision was accurate (with a score called IoU of 0.47 ± 0.02).
When the drone was constantly broken (blurry images and focus issues) and had no help, its vision dropped significantly to 0.28 ± 0.08.
With the basic self-adapting system, the robot managed to recover some of its vision, bringing the score back up to 0.30 ± 0.09.

However, the paper points out a flaw in their simple baseline: the robot was a bit clumsy. It often chose the "nuclear option" (redeploying the whole system) when a simple restart or setting change would have worked. This resulted in a high number of unnecessary redeploys (6.52 ± 4.67 per run) and a reaction time of about 2.02 ± 1.04 seconds. The paper suggests that while the robot can adapt, it needs to get smarter about how it adapts to avoid overreacting.

Why This Matters

The authors aren't claiming they have solved the problem of robot self-healing forever. Instead, they have built a challenging new test track. Before SUNSET, it was hard to compare different self-adaptation systems because everyone tested them on different, easy scenarios. Now, researchers have a standard, difficult playground where they can inject multiple, confusing faults at once.

The paper concludes that SUNSET is a "novel and challenging exemplar." It proves that we can simulate realistic, messy failures in a ROS2-based system (a common software framework for robots) and that we can measure exactly how well a robot's self-fixing brain performs. It's a step toward robots that can fly through a storm, get confused, and figure out how to land safely on their own, without needing a human to press "reset."

The code for this entire simulation is available for anyone to download and play with, inviting the next generation of robot engineers to build smarter, more resilient machines.

Technical Summary: SUNSET - A Sensor-fUsioN based semantic SegmEnTation exemplar for ROS-based self-adaptation

Problem Statement

Robotic systems are increasingly deployed in dynamic, unmapped environments, necessitating self-adaptive approaches to handle runtime complexities. A significant challenge in Robotics Software Architecture-based Self-adaptive Systems (RSASS) is the ambiguity of failure root causes. Often, system failures are easily observable (e.g., degraded performance or outages), but their underlying causes—whether internal faults or external perturbations—remain unknown. Furthermore, robotic systems frequently encounter concurrent failures, where multiple faults occur simultaneously or as ripple effects, complicating the adaptation process.

Current RSASS research lacks robust exemplars (testbeds) capable of evaluating self-adaptation strategies under these specific conditions: (1) failure symptoms that do not map cleanly to a single root cause, and (2) concurrent failures with ambiguous origins. Existing benchmarks often focus on theoretical scenarios, sequential failures with clear causes, or specific use cases tailored to a single managing system, making it difficult to generalize and compare different self-adaptive approaches.

Methodology

To address these gaps, the authors present SUNSET, a ROS2-based exemplar implementing a Sensor fUsioN based Semantic sEgmentaTion pipeline. The system is designed to simulate realistic robotic perception tasks while allowing for rigorous, repeatable evaluation of self-adaptive controllers.

System Architecture

Managed System: A semantic segmentation pipeline running on a simulated UAV (using the CARLA simulator and the SynDrone dataset). It fuses data from RGB and depth cameras.
- Components: Camera sensor node, Depth sensor node, Image enhancement node, Sensor fusion node, and a Segmentation node.
- ML Model: A ResNet50-based Deeplab-V3 model trained on the SynDrone dataset. Separate models are trained for different input modalities (RGB, Depth, RGB+Depth).
- Data Handling: The system uses pre-recorded ROS bag files to ensure experimental replicability.
Managing System: An external controller (separated from the managed system) responsible for detecting failures and executing adaptations. The paper provides a baseline managing system based on the MAPE-K feedback loop.
Fault Injection: A dedicated fault injector introduces 11 distinct faults directly into ROS nodes.

Faults and Failures

The exemplar simulates 11 faults that result in 5 observable failures (F1–F5). Crucially, multiple distinct faults can trigger the same failure symptom, requiring root cause analysis.

F1–F3 (Self-Healing): Communication channel outages caused by hardware disconnects, driver issues, memory leaks, synchronization errors, or GPU failures. These result in the same symptom (topic frequency drop) regardless of the specific fault.
F4 (Self-Optimization): Reduced segmentation performance (increased model entropy) caused by degraded image quality, incorrect image enhancement, sensor misalignment, or Gaussian noise in depth images.
F5 (Self-Optimization): Blurred camera images due to incorrect focus.

Adaptation Capabilities

SUNSET supports the four main adaptation types common in RSASS:

Reparametrization: Adjusting node parameters (e.g., triggering autofocus or recalibration).
Change of Communication: Switching subscribed topics at runtime.
Addition/Removal of Components: Restarting nodes via lifecycle state changes.
Redeploy: Terminating and restarting the OS process (simulating a hardware restart).

Key Contributions

The paper claims the following contributions:

Ambiguity Handling: An exemplar where multiple distinct faults lead to the same failure, necessitating root cause analysis by the managing system.
Concurrent Failure Support: The first ROS-based exemplar allowing managing systems to resolve concurrent failures using all common architectural adaptations (self-healing and self-optimization).
Realistic Performance Degradation: A trained ML model that produces realistic data degradation effects (e.g., entropy increases) when inputs are perturbed, rather than synthetic or arbitrary failure signals.
Reproducibility: Detailed documentation, fault-injection scripts, a baseline controller, and step-by-step integration guides to facilitate reproducible studies. The code is publicly available.

Evaluation and Results

The authors evaluated a baseline managing system (MAPE-K based) across 54 scenarios containing all possible combinations of simultaneous faults.

Metrics:

$f_{resolved}/s_{executed}$ : Ratio of resolved failures to executed adaptations.
$t_{react}$ : Time to detect symptoms and select the correct adaptation.
#redeploy $_u$ : Number of unnecessary redeploys (where a cheaper adaptation would have sufficed).
$t_{down}$ : System downtime (accumulated time without data).
IoU: Intersection over Union (segmentation quality).

Findings:

Baseline Performance: Without faults, the system achieved an IoU of $0.47 \pm 0.02$ . With persistent faults (F4 and F5) and no adaptation, IoU dropped to $0.28 \pm 0.08$ .
Adaptation Efficacy: The baseline managed to resolve 89% of failures ( $0.89 \pm 0.16$ ) but exhibited a high number of unnecessary redeploys ( $6.52 \pm 4.67$ ) because it defaulted to restarting nodes for F1–F3 without distinguishing the specific fault.
Uncertainty: The baseline failed to detect three out of four faults leading to F4 (performance degradation), resulting in a high standard deviation for IoU ( $0.30 \pm 0.09$ ) and highlighting the difficulty of distinguishing between different causes of performance drops.

Significance and Claims

The authors position SUNSET as a novel and challenging benchmark for the RSASS community. Its significance lies in:

Bridging the Gap: It fills the void of exemplars that evaluate self-adaptivity under concurrent, cause-ambiguous failures, a scenario prevalent in real-world robotics but under-represented in current literature.
ROS2 Standardization: Unlike many related works based on ROS1, SUNSET is built on ROS2, aligning with current industry standards and enabling the use of modern lifecycle management features.
Comprehensive Adaptation: It is the first exemplar to support the full spectrum of RSASS adaptations (reparametrization, communication changes, component addition/removal, and redeployment) within a single, concurrent failure context.

The paper concludes that SUNSET enables robotics research to evaluate managing systems with enhanced capabilities on an independent, realistic platform, moving beyond tailored use cases toward generalizable self-adaptive solutions.