Search-based Robustness Testing of Laptop Refurbishing Robotic Software

Technical Summary: Search-Based Robustness Testing of Laptop Refurbishing Robotic Software

Problem Statement
The Danish Technological Institute (DTI) employs collaborative robots (UR10e) to automate the refurbishment of laptops, a critical process for reducing electronic waste and supporting the Circular Economy. A key component of this workflow is the perception software, which utilizes Deep Learning (DL) object detection models (specifically YOLO variants) to identify screws, screw holes, and fixtures for disassembly and reassembly.

Current industrial practices for ensuring robustness rely on ad hoc data augmentation (e.g., flipping, blurring) during training. These methods lack a structured, automated mechanism to systematically assess model robustness or identify subtle failure-inducing conditions, such as specific lighting variations or surface wear. Incorrect detection can lead to physical damage to the laptops (e.g., scratching surfaces while removing stickers) or incomplete disassembly. The paper addresses the need for a systematic approach to uncover minimal, localized perturbations that expose weaknesses in these object detection models without causing catastrophic, unrealistic input corruption.

Methodology: PROBE
The authors propose PROBE, a search-based robustness testing approach that leverages multi-objective optimization to identify minimal, localized perturbations.

• Core Mechanism: PROBE treats robustness testing as a multi-objective optimization problem. It uses the NSGA-II (Non-dominated Sorting Genetic Algorithm II) algorithm to explore the perturbation space.
• Perturbation Encoding: Perturbations are encoded as seven-dimensional vectors representing a localized Gaussian modification applied to the image. The parameters include the center coordinates $(c_x, c_y)$, radius $r$, Gaussian spread ratio $\sigma$, and additive color shifts for Blue, Green, and Red channels ($\Delta B, \Delta G, \Delta R$).
• Search Constraints: The search is restricted to Regions of Interest (ROIs) defined by ground-truth bounding boxes to reduce computational cost and ensure relevance. Perturbation magnitudes are bounded to ensure changes remain subtle and realistic (e.g., color shifts limited to $[-48, 48]$).
• Objective Functions: The algorithm minimizes a vector of three objectives:
  1. Confidence Degradation ($f_1$): Reducing the average confidence of correct (same-class) detections.
  2. Localization Degradation ($f_2$): Reducing the Intersection over Union (IoU) of the best same-class prediction.
  3. Perturbation Budget ($f_3$): Minimizing the magnitude and spatial extent of the perturbation to ensure it remains minimal.
• Failure Definition: A failure is identified if the perturbed input results in missed detections, misclassifications, mislocalizations (IoU below a threshold), or ambiguous predictions.
• Metamorphic Analysis: Beyond explicit failures, PROBE assesses model stability under non-failing perturbations using Metamorphic Relations (MRs) to measure deviations in confidence and localization scores.

Experimental Setup
The approach was evaluated on three industrial object detection models provided by DTI, trained on progressively complex datasets:

1. origSCDM: Trained on black screws only.
2. noscrewSCDM: Trained on screws and non-screw objects.
3. fixtureSCDM: Trained on screws, non-screws, and diverse fixtures with extensive augmentations.

The evaluation compared PROBE against a Random Search (RS) baseline using a Hypervolume (HV) metric, failure rates, perturbation magnitudes, and cross-model transferability. The study utilized 802 images for evaluation and 25 disjoint images for transferability testing.

Key Results

• Search Effectiveness: PROBE significantly outperformed Random Search. It generated failure-inducing perturbations at rates between 10.4% and 26.3%, compared to 1.5% to 9.9% for RS.
• Search Quality: PROBE achieved consistently higher Hypervolume (HV) scores across all models, indicating superior exploration of the trade-offs between failure induction and perturbation magnitude.
• Perturbation Magnitude: PROBE identified failures using smaller perturbation magnitudes than RS, suggesting it finds more subtle, realistic vulnerabilities.
• Transferability: Perturbations generated by PROBE transferred effectively across different models (e.g., perturbations for noscrewSCDM induced failures in origSCDM at a 29.8% rate vs. 16.0% for RS).
• Failure Patterns:
  • origSCDM and noscrewSCDM failures were dominated by misclassifications and ambiguous predictions.
  • fixtureSCDM failures were primarily driven by mislocalization errors, and the model exhibited the highest overall robustness.
• Stability Analysis: Under non-failing perturbations, fixtureSCDM showed high stability (only ~1% confidence violations and near-zero localization violations), whereas the other models showed higher rates of moderate to large deviations.

Significance and Contributions
The paper claims the following contributions:

1. Systematic Robustness Testing: PROBE provides a structured, automated alternative to ad hoc data augmentation, enabling the systematic discovery of failure-inducing conditions in industrial robotic software.
2. Minimal Perturbation Discovery: By optimizing for minimal changes, PROBE uncovers subtle vulnerabilities that are more likely to occur in real-world environments (e.g., varying lighting) compared to gross corruptions.
3. Cross-Model Insights: The study demonstrates that failure-inducing perturbations can generalize across model versions, suggesting that testing efforts can be reused to improve testing efficiency in iterative development cycles.
4. Metamorphic Robustness Assessment: The work extends robustness evaluation beyond binary failure detection, using MRs to quantify model stability even when predictions remain formally correct.
5. Industrial Relevance: The approach is validated on real-world laptop refurbishment workflows, highlighting the specific challenges of operating in unstructured environments with high device variability.

Limitations and Future Work
The authors acknowledge that the study is limited to three specific YOLO models from a single industrial case study, and generalizability to other domains or architectures (e.g., Faster R-CNN) requires further investigation. Future work aims to integrate PROBE into continuous testing pipelines (CI/CD) to feed discovered failures back into training, extend the approach to other perception components (e.g., sticker detection), and explore self-adaptive recovery methods for robotic software.