💻 computer science

UAMTERS: Uncertainty-Aware Mutation Analysis for DL-enabled Robotic Software

This paper presents UAMTERS, an uncertainty-aware mutation analysis framework that introduces stochastic uncertainty injection operators and specialized metrics to effectively evaluate the dependability and test suite quality of deep learning-enabled robotic software in dynamic environments.

Original authors: Chengjie Lu, Jiahui Wu, Shaukat Ali, Malaika Din Hashmi, Sebastian Mathias Thomle Mason, Francois Picard, Mikkel Labori Olsen, Thomas Peyrucain

Published 2026-02-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chengjie Lu, Jiahui Wu, Shaukat Ali, Malaika Din Hashmi, Sebastian Mathias Thomle Mason, Francois Picard, Mikkel Labori Olsen, Thomas Peyrucain

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have built a very smart robot, like a robotic arm that can peel stickers off laptops or a robot that helps disassemble computers. This robot uses "Deep Learning" (a type of AI) to see the world, make decisions, and move its arms. It's like giving the robot a brain that learned by looking at thousands of photos.

But here's the problem: AI brains are a bit jittery.

Sometimes, if the lighting is weird, or if the robot sees something it hasn't seen before, the AI might get confused. It might think a sticker is a piece of tape, or it might miss a screw entirely. This "jitteriness" is called uncertainty. In the real world, where things change constantly, this uncertainty can cause the robot to make mistakes, which is dangerous if the robot is working with expensive equipment or people.

So, how do we make sure the robot is safe? We need to test it. But testing a robot isn't like testing a toaster; you can't just flip a switch. You need to check if the robot's "test suite" (a collection of test scenarios) is good enough to catch these jittery mistakes.

This is where the paper UAMTERS comes in. Think of it as a new, super-smart way to stress-test a robot's brain.

The Old Way vs. The New Way

The Old Way (Traditional Mutation Analysis):
Imagine you want to see if a security guard is good at spotting intruders. The old method is to hire an actor to sneak into the building wearing a mask. If the guard catches them, the guard passes. If the guard misses them, the guard fails.

The Flaw: In the world of AI, the "intruder" (the mistake) isn't always a clear mask. Sometimes the mistake happens because the AI is just uncertain. The old tests didn't really check if the guard could spot mistakes caused by confusion; they only checked if the guard could spot clear-cut errors.

The New Way (UAMTERS):
The authors of this paper say, "Let's simulate the AI getting confused on purpose!"
They created a framework called UAMTERS (Uncertainty-Aware Mutation Analysis for Robotic Software).

Here is how it works, using a creative analogy:

1. The "Confusion Injectors" (Mutation Operators)

Imagine the robot's brain is a team of 100 experts trying to solve a puzzle together.

The MCD Operator (The "Napping Experts"): This method randomly tells some experts to take a nap (drop them out) while the team is working. If too many experts are napping, the team might make a bad guess. This simulates the AI being unsure because it's missing some information.
The MCB Operator (The "Blindfolded Experts"): This method puts blindfolds on whole groups of experts at once. If a whole section of the team can't see, the team's guess will be wild. This simulates the AI being unsure because the image is blurry or noisy.

By doing this, the researchers create "Mutants"—versions of the robot's brain that are intentionally a bit "drunk" or "confused."

2. The "Scorecard" (Mutation Scores)

Now, they run these confused robot brains through a series of tests (like a sticker-removal task).

The Old Scorecard: It just asked, "Did the robot fail?" (Yes/No).
The UAMTERS Scorecard: This is much more detailed. It asks:
- "Did the robot miss a sticker it should have seen?" (Like a blind spot).
- "Did the robot see a sticker that wasn't there?" (Like a hallucination).
- "Did the robot guess the sticker's location slightly wrong?" (Like being off by a few millimeters).
- "How confused was the robot when it made the mistake?"

This new scorecard gives a much clearer picture of how the robot fails when it's uncertain.

Why Does This Matter?

The researchers tested this on three real-world robots:

A robot that peels stickers off laptops.
A robot that takes laptops apart.
A robot that recognizes human faces.

They found that the old tests were often blind to the robot's confusion. They would say, "The test suite is perfect!" even though the robot was failing when the lighting changed.

But the UAMTERS tests were like a high-powered microscope. They could see exactly which test scenarios were good at catching the robot when it got confused, and which ones let the robot slide by with dangerous mistakes.

The Big Takeaway

Think of UAMTERS as a "Stress Test for Uncertainty."

Just as you wouldn't build a bridge without testing how it handles strong winds, you shouldn't let a robot work in a dynamic world without testing how it handles confusion. This paper gives engineers a new toolkit to:

Inject confusion into the robot's brain safely.
Measure exactly how well their tests catch those confused moments.
Build better robots that are safer and more reliable, even when things don't go exactly as planned.

In short, UAMTERS helps us stop asking, "Did the robot pass the test?" and start asking, "Did the robot pass the test even when it was confused?"

1. Problem Statement

Self-adaptive robots operating in dynamic, unpredictable environments increasingly rely on Deep Learning (DL) components for perception, decision-making, and control. However, DL models inherently exhibit uncertainty (e.g., due to limited training data, sensor noise, or stochastic inference mechanisms like dropout). This uncertainty can lead to unpredictable failures that traditional testing methods often miss.

While mutation analysis is a standard technique for evaluating test suite effectiveness by injecting faults, existing approaches for DL-enabled software (e.g., DeepMutation, DeepCrime) primarily focus on structural faults or training-time regularization. They fail to explicitly model predictive uncertainty during inference. Consequently, there is a lack of techniques to assess whether test suites can effectively detect failures caused specifically by the stochastic nature of DL models in robotic systems.

2. Methodology: UAMTERS Framework

The authors propose UAMTERS (Uncertainty-Aware Mutation Analysis for DL-enabled Robotic Software), a framework designed to inject stochastic uncertainty into DL models and evaluate test suites based on their ability to detect the resulting behavioral changes.

A. Uncertainty-Aware Mutation Operators

Instead of modifying weights or architecture permanently, UAMTERS injects uncertainty at inference time using two novel operators based on Bayesian approximation methods:

MC-Dropout-based Mutation (MCD): Enables dropout layers during inference. A mutation ratio ( $\mu$ ) controls the probability of dropping neurons, simulating aleatoric uncertainty.
MC-DropBlock-based Mutation (MCB): Designed for Convolutional Neural Networks (CNNs) used in object detection. It drops contiguous blocks of neurons in feature maps. It uses two parameters: dropout rate ( $\mu_d$ ) and block size ( $\mu_b$ ), allowing control over both the intensity and spatial extent of injected uncertainty.

B. Novel Mutation Score Metrics

To evaluate the effectiveness of test suites in detecting these uncertainty-induced failures, the authors propose three categories of mutation scores tailored for object detection tasks (the primary focus of their case studies):

Image-Level Mutation Scores (Baseline): Traditional metrics determining if a mutant is "killed" (output differs from the original) based on statistical significance across multiple runs.
Object-Level Mutation Scores: Granular metrics analyzing specific detection outcomes:
- Miss-based ( $MS_{miss}$ ): Measures the proportion of objects correctly detected by the original model but missed by the mutant (False Negatives).
- Ghost-based ( $MS_{ghost}$ ): Measures objects detected by the mutant but not the original (False Positives).
- Combined ( $MS_{mg}$ ): Aggregates both misses and ghosts.
IoU-based Mutation Score: Quantifies the degradation in spatial overlap (Intersection over Union) for objects that are still detected by both models but with shifted boundaries.
Uncertainty-Aware Mutation Scores (UA-MS): The core contribution. These scores measure the difference in predictive uncertainty between the original model and the mutant. They utilize Uncertainty Quantification (UQ) metrics:
- Classification: Variation Ratio (VR), Shannon Entropy (SE), Mutual Information (MI).
- Regression (Bounding Boxes): Total Variance (TV), Predictive Surface (PS).
- These are calculated for Match, Miss, and Ghost object sets, resulting in 15 distinct UA-MS metrics.

3. Experimental Setup

The framework was evaluated across three industrial-level robotic case studies involving seven DL models and eight test suites:

Laptop Sticker Removing Robot (LSRR): Uses Faster R-CNN, RetinaNet, and SSD variants for sticker detection.
Laptop Disassembling Robot (LDR): Uses YOLOv11 for screw localization.
TIAGo Pro: A mobile manipulator using YuNet for human face detection.

The experiments varied mutation ratios (dropout rates from 0.10 to 0.50 and block sizes) to simulate increasing levels of uncertainty. Statistical tests (Kruskal-Wallis, Spearman's correlation, Multiple Correlation) were used to analyze results.

4. Key Results

The study addresses three Research Questions (RQs):

RQ1: Comparison with Traditional Scores:
- Traditional Image-Level Mutation Scores (Img-MS) often failed to distinguish between test suites (e.g., yielding 100% kill rates for all suites, providing no differentiation).
- The proposed Object-Level (Obj-MS) and Uncertainty-Aware (UA-MS) scores successfully distinguished test suite quality, showing clear trends where test suites with fewer objects or specific characteristics were more effective at revealing uncertainty-induced failures.
RQ2: Effectiveness in Distinguishing Test Suites:
- Statistical analysis (Kruskal-Wallis) confirmed that Obj-MS (especially $MS_{miss}$ and $MS_{mg}$ ) and IoU-MS consistently showed statistically significant differences ( $p < 0.01$ ) and large effect sizes across all models.
- Among UA-MS, metrics derived from the Match set ( $MS_{match}^{vr, ie, mi}$ ) and Miss set were the most stable and effective in highlighting differences between test suites.
RQ3: Sensitivity to Uncertainty Levels:
- Obj-MS and IoU-MS showed a very high positive correlation ( $\rho \approx 1.0$ ) with increasing mutation ratios (dropout rates), confirming they effectively capture model degradation as uncertainty increases.
- UA-MS metrics generally correlated well with uncertainty levels, though some metrics (specifically those related to "Ghost" objects in certain models) showed irregular patterns, suggesting that false positives driven by noise are harder to quantify linearly.
- MCB Operator: The block size parameter was found to have a significant impact, often creating steeper gradients in mutation scores than dropout rate alone.

5. Key Contributions

UAMTERS Framework: A systematic approach for mutation analysis in DL-enabled robotics that explicitly injects stochastic uncertainty at inference time.
Novel Mutation Operators: MCD and MCB operators that simulate predictive uncertainty rather than just structural faults.
New Mutation Metrics: 19 novel scores (including UA-MS) that quantify a test suite's ability to detect failures caused by uncertainty, moving beyond simple "pass/fail" binary outcomes.
Empirical Validation: Extensive evaluation on seven industrial models demonstrating that traditional metrics are insufficient for uncertainty-aware testing, while the proposed metrics provide sensitive and reliable assessments.

6. Significance and Impact

Safety-Critical Assurance: The work addresses a critical gap in validating self-adaptive robots. By focusing on uncertainty-induced failures, UAMTERS helps ensure that robotic systems can handle unpredictable real-world scenarios safely.
Integration with MAPE-K: The framework directly supports the "Legitimate" component of the MAPLE-K loop (a self-adaptation framework), enabling robots to validate planned actions against uncertainty before execution.
Guidance for Test Generation: The metrics can guide the generation of better test suites and the prioritization of tests that are most likely to expose uncertainty-related vulnerabilities.
Future Directions: The authors plan to extend these techniques to Vision-Language-Action (VLA) models and integrate them into continuous testing pipelines for real-time robot assessment.

In summary, UAMTERS shifts the paradigm of mutation analysis for robotic software from detecting static faults to evaluating resilience against the inherent stochasticity of Deep Learning, providing a more robust foundation for deploying autonomous systems in dynamic environments.