When Can We Trust Deep Neural Networks? Towards Reliable Industrial Deployment with an Interpretability Guide

1. Problem Statement

Deep learning models, particularly in safety-critical industrial domains (e.g., defect inspection, autonomous driving, medical diagnosis), suffer from a lack of reliability. Despite achieving high accuracy, these models operate as "black boxes" without internal mechanisms to flag unreliable predictions.

• The Core Vulnerability: Existing methods like uncertainty estimation or confidence thresholds are effective at detecting out-of-distribution (OOD) data but fail against "in-distribution yet confidently wrong" predictions.
• The Risk: A single undetected False Negative (FN) (e.g., missing a critical defect) can lead to catastrophic outcomes. Current paradigms rely on a linear Data → Model → Output pipeline, leaving users with no way to verify if the model's reasoning aligns with its output.

2. Methodology

The authors propose a post-hoc explanation-based framework to detect suspicious samples (potential errors) without retraining the model or modifying its architecture. The core hypothesis is that the reliability of a prediction can be judged by comparing different types of internal visual explanations.

A. Theoretical Basis: Discriminative vs. Class-Agnostic Explanations

The method leverages the discrepancy between two types of visual heatmaps generated for the same input:

1. Discriminative Explanation (Class-Specific): Uses Grad-CAM. It highlights features specific to the predicted class.
2. Class-Agnostic Explanation (Non-Discriminative): Uses FullGrad. It aggregates gradient information from all layers (including biases) to highlight all non-background features, regardless of the specific class.

Key Observation:

• True Negatives (TN): In defect-free images, Grad-CAM (focused on the "defect" class) and FullGrad (focused on "any feature") produce significantly different heatmaps because there are no defect features to highlight.
• False Negatives (FN): In images with undetected defects, both methods tend to highlight the same defect regions (as the model fails to classify them, but the features exist), resulting in high similarity between the heatmaps.

B. The $\Delta$-IoU Indicator

To quantify this discrepancy, the authors define a metric called $\Delta$-IoU:

1. Generate heatmaps for the positive class ($E_{d}^{c1}$), negative class ($E_{d}^{c0}$), and the class-agnostic view ($E_{n}$).
2. Binarize the heatmaps (using Otsu thresholding).
3. Calculate the Intersection over Union (IoU) difference:
   $$\Delta IoU = IoU(E_{d}^{c1}, E_{n}) - IoU(E_{d}^{c0}, E_{n})$$
4. Logic: A low $\Delta$-IoU indicates that the class-specific explanation aligns closely with the class-agnostic one, suggesting the model is "confused" or missing a feature it should have detected. Samples with anomalous $\Delta$-IoU are flagged as Suspicious Samples.

C. Adversarial Enhancement

To ensure 100% recall (detecting all false negatives), the authors introduce an Adversarial Enhancement step:

• Before calculating $\Delta$-IoU, the input image is iteratively perturbed to shift features toward the positive (defect) class.
• Formula: $x_t = x_{t-1} + \alpha \frac{\partial y_{c1}}{\partial x_{t-1}}$
• This amplifies subtle defect features that the original model might have ignored, making the discrepancy between Grad-CAM and FullGrad more pronounced for FN samples.

3. Key Contributions

1. First Proactive Error Detection: Proposes the first criterion to proactively identify potentially erroneous network outputs based on internal reasoning discrepancies, rather than just output confidence.
2. Training-Free Approach: The method requires no architectural changes or retraining; it is a post-hoc analysis tool.
3. New Paradigm: Advocates for shifting the standard AI pipeline from Data-Model-Output to Data-Model-Interpretation-Output, integrating interpretability directly into the decision-making loop.
4. Adversarial Enhancement: Introduces a technique to boost the detection of subtle defects, achieving near-perfect recall in safety-critical tasks.

4. Experimental Results

The method was evaluated on two industrial defect detection benchmarks: Kolektor SDD and Kolektor SDD2.

• Baseline Performance: Standard confidence-based thresholds failed to detect most false negatives (e.g., only identified 2/19 risks in SDD2).
• $\Delta$-IoU Performance:
  • On Kolektor SDD, the method achieved 100% Recall (0 FNs) with 100% accuracy, though it flagged some True Negatives as suspicious (conservative approach).
  • On Kolektor SDD2, it improved recall from 82.7% (Origin) to 91.0% and accuracy to 98.3%.
• With Adversarial Enhancement ($\Delta$-IoU$_{adv}$):
  • On the CBAM-enhanced VGG16 model, the method achieved 100% Recall (0 FNs remaining).
  • Trade-off: Achieving 100% recall required flagging a significant number of True Negatives as suspicious (False Positives in the detection task), dropping from 881 TNs to 250 TNs. This is acceptable in safety-critical contexts where missing a defect is worse than a false alarm.

5. Significance and Limitations

• Significance: The paper provides a practical solution for deploying AI in safety-critical industries by adding a "safety layer" based on interpretability. It proves that internal feature discrepancies can serve as a reliable proxy for model failure.
• Limitations:
  • False Positives: The method currently flags many True Negatives as suspicious (high false positive rate in the detection task), which may reduce throughput in high-speed industrial lines.
  • Feature Origins: The exact origin of the discriminative features (shallow vs. deep layers) requires further investigation.
  • Domain Knowledge: The current approach lacks specific domain knowledge integration, which contributes to the misclassification of visually ambiguous samples.

Conclusion: The authors successfully demonstrate that integrating interpretability into the deployment pipeline is essential for reliable AI. By detecting the "confidently wrong" failures that traditional metrics miss, their method lays the foundation for trustworthy AI in mission-critical applications.