Empirical Insights of Test Selection Metrics under Multiple Testing Objectives and Distribution Shifts

Technical Summary: Empirical Insights of Test Selection Metrics under Multiple Testing Objectives and Distribution Shifts

1. Problem Statement

Deep Learning (DL)-based systems are increasingly deployed in safety-critical domains (e.g., autonomous driving, malware detection). However, these models often exhibit unexpected misbehavior when encountering out-of-distribution (OOD) scenarios. To ensure reliability, researchers use test selection metrics to pick a small, representative subset of data (a test suite) for manual labeling and evaluation.

The authors identify three critical gaps in existing research:

• Narrow Testing Objectives: Most studies only evaluate metrics for fault detection, leaving their effectiveness for performance estimation or retraining guidance unclear.
• Limited OOD Coverage: Prior work focuses heavily on original data or simple corrupted/adversarial shifts, largely ignoring natural and label shifts.
• Dataset Bias: Evaluations are overwhelmingly dominated by image data, with text and Android application data remaining underexplored.

2. Methodology

The authors conducted a massive empirical study involving 1,640 experimental scenarios to provide a unified benchmark.

• Metrics Evaluated: 15 existing metrics categorized into six types: Uncertainty-based, Diversity-based, Surprise-based, Sampling-based, Clustering-based, and Hybrid.
• Testing Objectives:
  1. Fault Detection: Maximizing the number of mispredictions (#Mis.) and unique fault clusters (#Clu.).
  2. Performance Estimation: Minimizing the Absolute Error (AE%) between the test suite accuracy and the full dataset accuracy.
  3. Retraining Guidance: Maximizing the accuracy improvement (Acc.%) of a model after retraining on the selected suite.
• OOD Scenarios: Five types of distribution shifts: Corrupted, Adversarial, Temporal, Natural, and Label shifts.
• Data Modalities & Models: Three modalities (Images, Text, and Android packages) across 13 different DL models (including CNNs, LSTMs, Transformers, etc.).
• Experimental Design: The study varied selection budgets (50 to 200 samples) and compared two retraining processes (Type I: retraining on selected samples only; Type II: retraining on original training data + selected samples).

3. Key Contributions

• The First Unified Benchmark: A systematic assessment of 15 metrics across three objectives, five OOD types, and three modalities.
• Validation of Fault Clustering: Proved that high-quality clustering pipelines (e.g., using UMAP and DBSCAN) can reliably represent distinct DNN faults through feature pattern inspection and cluster-specific retraining.
• Retraining Strategy Insights: Identified that the optimal retraining process (Type I vs. Type II) depends on the data modality (e.g., Type II is better for dense visual data, while Type I is better for sparse Android data).

4. Key Results and Findings

• Objective-Metric Mismatch:
  • Performance Estimation: Surprisingly, metrics designed for this objective (like CES or EST) were outperformed by simple diversity-based metrics (GD, STD) and even Random sampling.
  • Fault Detection: DSA (Surprise-based) is highly effective for classification, while NC (Diversity-based) is superior for regression tasks.
  • Retraining Guidance: KMNC and PACE provided the highest accuracy improvements, outperforming metrics specifically designed for retraining (MCP, DAT).
• OOD Robustness: The relative performance rankings of metrics are remarkably stable across different OOD scenarios. A metric that works well on ID data generally maintains its ranking under various shifts.
• Budget & Efficiency:
  • Stability: Metrics for fault detection and performance estimation are stable across different budgets, whereas retraining guidance performance fluctuates significantly.
  • Trade-offs: A clear performance-speed trade-off exists for fault detection (the best metric, DSA, is computationally expensive, while uncertainty metrics are faster but less effective). However, no such trade-off was found for performance estimation or retraining.

5. Significance

This paper provides a "gold standard" guide for practitioners in the software engineering and AI communities. Instead of relying on the original design intent of a metric, testers can now choose metrics based on their actual performance across diverse real-world conditions. It offers actionable recommendations, such as using diversity-based metrics for accurate performance estimation and specific clustering pipelines for targeted model repair in different data modalities.