A New Kind of Adversarial Example: Measuring the Human-Model Gap, and Its Relationship to OOD Detection

Technical Summary: A New Kind of Adversarial Example (NKE)

Problem Statement
Classical adversarial examples are defined by small, imperceptible perturbations that cause a model to misclassify an image while remaining visually unchanged to a human. This paper investigates the inverse failure mode: New Kind of Adversarial Examples (NKE). In this setting, a large, clearly visible perturbation is applied to an image such that the model maintains its original, correct prediction with high confidence, even though a human observer would no longer be able to recognize the image.

While prior work (Borji [3]; Nie et al. [5]) established that such examples can be generated at scale, three critical questions remained untested:

1. The Human–Model Gap: Does human recognition actually degrade significantly faster than model accuracy under these large perturbations, or is the gap merely an artifact of signal loss?
2. OOD Detection and Calibration: Do standard Out-of-Distribution (OOD) detectors and calibration tools (e.g., confidence scores, energy-based methods) identify these examples as failures, given they are far from the data manifold yet confidently classified?
3. Robustness: Do existing defenses (adversarial training, input transformations, randomized smoothing) mitigate this failure mode, or is resistance to small-$\epsilon$ attacks orthogonal to resistance to large-$\epsilon_l$ attacks?

Methodology
The study utilizes a unified attack generation pipeline to produce NKE images across MNIST, CIFAR-10, and ImageNet (using Imagenette). The attack minimizes the true-label loss (keeping the prediction correct) while enforcing a lower-bound constraint on the perturbation magnitude ($D(x, x') > \epsilon_l$). The primary generator uses momentum-based iterative variants (NMI-FGSM/NMI-FGM).

The evaluation is organized into four tracks:

1. Human–Model Gap Measurement:
   • Computational Proxies: An independent generic recognizer (a different architecture trained on clean data) and a shape recognizer (trained on Canny edge maps) serve as proxies for human performance.
   • Human Pilot: A small-scale forced-choice study ($N=5$) using a released harness to collect real human responses.
   • Controls: Matched-magnitude Gaussian noise controls are used to distinguish between "adversarial structure" effects and generic "signal loss."
   • CLIP Proxy: Zero-shot CLIP classification is used as a stronger, non-CNN proxy for human recognizability at ImageNet scale.
2. OOD and Calibration Analysis: Evaluation of Maximum Softmax Probability (MSP), Energy scores, and Mahalanobis distance detectors, alongside Expected Calibration Error (ECE).
3. Adaptive Attack: Testing whether feature-space detectors can be evaded by an attacker who incorporates the detector's distance metric into the optimization objective.
4. Defense Evaluation: Assessing adversarial training, randomized smoothing, and input transformations against NKE, and correlating classical robustness with NKE resistance.
5. Mechanistic Analysis: Quantifying the preservation of edge/shape structure versus texture (via GLCM and Canny edge F1 scores) to explain the model's invariance.

Key Results

• The Human–Model Gap is Real and Structural:

  • On CIFAR-10, as $\epsilon_l$ increases, the source model maintains 100% accuracy, while an independent generic recognizer drops to ~49%.
  • Crucially, a matched-L2 Gaussian control degrades the recognizer faster (to ~15%) than the structured NKE perturbation. This indicates the gap is not due to signal loss but to the specific adversarial structure that preserves model-relevant features while destroying human-recognizable content.
  • A small human pilot ($N=5$) corroborates the gap, with human accuracy dropping to ~8% at large perturbations while the model remains at 100%.
  • At ImageNet scale, a CLIP zero-shot proxy confirms a significant gap (model 100% vs. CLIP ~33%), though the ordering of NKE vs. Gaussian noise reverses compared to CIFAR-10, suggesting NKE actively destroys broad semantic content at this scale.

• Blindness of Standard OOD and Calibration Tools:

  • Confidence-based (MSP) and energy-based OOD detectors flag 0% of NKE images, and ECE remains near 0. The models appear perfectly calibrated despite the input being unrecognizable to humans.
  • Feature-space Mahalanobis detectors flag 100% of NKE images, correctly identifying them as off-manifold.
  • Adaptive Vulnerability: However, an adaptive attacker can reduce Mahalanobis detection from 100% to 0% by adding a penalty term to the loss function, with no cost to attack success or perturbation magnitude. This evasion holds against held-out detectors and adversarially trained models.

• Orthogonality to Classical Defenses:

  • No classical defense mitigates NKE. Adversarially trained models (with 45% robust accuracy against PGD) remain ~100% vulnerable to NKE.
  • There is zero correlation ($r \approx 0$) between resistance to small-$\epsilon$ attacks and resistance to large-$\epsilon_l$ NKE attacks.

• Mechanistic Insight (Texture vs. Shape):

  • The attack destroys low-level texture statistics (GLCM similarity collapses to ~0) much faster than it degrades edge/shape structure (edge preservation remains ~0.3–0.4).
  • While CNNs are texture-biased, a Vision Transformer (ViT) with a shape bias does not recover the label better than a CNN on NKE inputs, indicating the surviving structure is insufficient for human-like recognition.
  • Transferability analysis shows NKE examples are highly model-specific; as task complexity increases (MNIST $\to$ ImageNet), transferability collapses, suggesting the surviving signal is idiosyncratic to the source model.

Significance and Claims
The paper claims to move the discussion from the possibility of constructing such examples to understanding their implications for the gap between model and human perception and the limitations of current safety tools.

1. Reframing the Failure Mode: NKE represents a distinct vulnerability where models are "confidently wrong" in the human sense (unrecognizable) but "correct" in the model sense. This is a failure mode that standard confidence-based monitoring is structurally blind to.
2. Limitations of Current Defenses: The findings demonstrate that robustness to small perturbations does not imply robustness to large, label-preserving perturbations. Existing defenses do not address this regime.
3. OOD Detection Nuance: While feature-space detectors can identify NKE examples, they are not adaptively robust. The paper suggests that "off-manifold" detection is necessary but insufficient once the detector is known to the attacker.
4. Human–Model Divergence: The study provides empirical evidence (via proxies and a pilot) that the divergence between human and model psychometric curves is not just a theoretical concept but a measurable reality in large-perturbation regimes, specifically driven by the destruction of texture while preserving shape-like structures that models rely on.

The authors conclude that systems relying on softmax confidence or energy scores for distributional failure detection should be audited against this specific regime, and that future work must focus on detectors that are adaptively robust by construction. The paper releases the full pipeline, stimuli, and a human-study harness to facilitate this auditing.