Defending against Patch-Based and Texture-Based Adversarial Attacks with Spectral Decomposition

1. Problem Statement

Deep Neural Networks (DNNs), particularly in object detection, are vulnerable to physically realizable adversarial attacks. Unlike digital attacks that add imperceptible noise, these physical attacks involve:

• Patch-based attacks: Placing a localized, high-contrast adversarial patch (e.g., on a T-shirt or a cardboard sign) to fool the detector.
• Texture-based attacks: Covering the entire surface of an object (e.g., adversarial clothing patterns) with a pervasive texture to hide the object.

Limitations of Existing Defenses:

• Adaptive Attacks: Most existing defenses (e.g., patch detectors, inpainting) fail when the attacker specifically designs the perturbation to bypass the defense (white-box setting).
• Scope: Many defenses target only patch-based attacks and struggle with texture-based attacks, which are more spatially pervasive and harder to localize.
• Frequency Domain Gaps: Existing frequency-based defenses (using DFT or JPEG compression) often focus on removing high-frequency noise but fail to handle the large, low-frequency color blocks characteristic of physical patches and textures.

2. Methodology: Adversarial Spectrum Defense (ASD)

The authors propose ASD, a defense mechanism that leverages Discrete Wavelet Transform (DWT) to analyze and mitigate adversarial patterns across multiple frequency scales. ASD is designed to work in conjunction with off-the-shelf Adversarial Training (AT) models.

A. Core Insight: Spectral Analysis

The authors analyzed the frequency components of physical adversarial examples using DWT. They found that:

• Both patch-based (AdvPatch) and texture-based (AdvTexture) attacks exhibit significantly higher amplitudes in the DWT detail coefficients (high-frequency sub-bands) compared to benign image patches.
• Unlike Discrete Fourier Transform (DFT), which captures global frequency, DWT provides localized frequency information, making it ideal for detecting localized adversarial patterns.

B. The ASD Pipeline

The defense consists of three main stages:

1. Multi-Spectral Adversarial Region Masking (ASD Masking):

   • Decomposition: The input image is converted to grayscale and decomposed using 2D DWT (using Haar wavelets) into multiple levels ($n$). Each level produces detail coefficients ($D_i$) representing different frequency bands.
   • Intensity Mapping: The $L_2$ norm of the detail coefficients across channels (horizontal, vertical, diagonal) is computed to create a spectral intensity map.
   • Detection & Smoothing: The intensity map is smoothed to reduce noise. Pixels exceeding a dynamic threshold ($\theta_i = 2^{i-1}\theta$) are identified as adversarial regions.
   • Mask Fusion: Masks from different decomposition levels are upsampled and fused (union operation) to create a final binary mask covering the adversarial regions.
   • Preprocessing: The detected adversarial regions in the original image are blurred (Gaussian blur) to suppress the high-amplitude perturbations before feeding the image to the detector.

2. ASD Level Aggregation:

   • Since object sizes vary (a person far away vs. close up), an adversarial pattern might appear in different frequency bands depending on the scale.
   • ASD runs the masking process with multiple decomposition levels ($n_1, n_2, \dots, n_k$).
   • The detection results from all levels are aggregated using Non-Maximum Suppression (NMS) to ensure robustness against objects of varying sizes.

3. Integration with Adversarial Training (AT):

   • ASD handles high-amplitude perturbations (physical patches/textures) by masking them.
   • However, an adaptive attacker might try to evade ASD by creating low-amplitude perturbations (constrained $L_\infty$ norm).
   • The authors prove that low $L_\infty$ perturbations inherently result in low DWT amplitudes. Therefore, ASD is paired with a standard $L_\infty$-Adversarial Training (AT) model.
   • Synergy: ASD removes high-amplitude threats; AT provides robustness against low-amplitude threats that ASD might miss.

3. Key Contributions

1. Novel Defense Strategy: Proposes ASD, the first defense to utilize multi-resolution spectral decomposition (DWT) specifically to detect and mask both localized patches and pervasive textures.
2. Comprehensive Framework: Integrates spectral analysis with off-the-shelf Adversarial Training, creating a defense effective against both high-amplitude (physical) and low-amplitude (digital-like) perturbations in an adaptive setting.
3. Theoretical Insight: Provides a mathematical proof linking $L_\infty$ perturbation bounds to DWT amplitude bounds, justifying the combination of ASD and AT.
4. State-of-the-Art Performance: Demonstrates superior performance across various detectors (Faster R-CNN, FCOS, YOLOv8) and attack types.

4. Experimental Results

The authors evaluated ASD+AT against strong adaptive white-box attacks (where the attacker knows the defense).

• Performance Gains:

  • Against a suite of 6 attacks (4 physical, 2 imperceptible), ASD+AT achieved an average AP50 of 77.11%, outperforming previous state-of-the-art defense methods by 14.47%.
  • Specifically against patch and texture attacks, it improved AP50 by 21.73% over the best baseline.
  • On the AdvPatch attack, ASD+AT achieved 90.04% AP (vs. ~55-60% for baselines).
  • On the AdvTexture attack, ASD+AT achieved 70.08% AP (vs. ~25-40% for baselines).

• Robustness to Adaptive Attacks:

  • Tested against three specific adaptive attack strategies designed to bypass ASD:
    1. Direct Gradient Attack: Optimizing input against the defended model.
    2. Loss-based Attack: Adding a loss term to minimize DWT coefficients.
    3. Range-based Attack: Constraining pixel values to keep DWT amplitudes low.
  • ASD+AT maintained high performance (AP > 88%) even against these tailored attacks, whereas standard AT alone dropped significantly (e.g., to 55.46% on AdvPatch).

• Physical World Validation:

  • Evaluated on real-world video footage of adversarial clothing. ASD+AT achieved 100% defense performance against both AdvTexture and AdvCaT attacks in non-adaptive physical settings.

• Ablation Studies:

  • Confirmed that removing AT causes a massive drop in performance (proving the need for low-amplitude defense).
  • Confirmed that using multiple decomposition levels (Level Aggregation) is crucial for detecting objects of different scales.
  • Showed that DWT outperforms FFT and DCT because DWT captures spatial localization.

5. Significance

• Security for Critical Systems: This work addresses a critical gap in the security of autonomous systems and surveillance, where physical adversarial attacks (wearing a "cloaking" shirt or holding a sign) pose real-world threats.
• Adaptive Resilience: By combining spectral analysis with adversarial training, the method remains effective even when attackers explicitly try to reverse-engineer the defense, a common failure point for previous methods.
• Efficiency: While adding some computational overhead (approx. 2.5x latency for full aggregation), the method is significantly faster than some frequency-based baselines (like FD-JPEG) and offers a massive gain in robustness.
• Generalizability: The approach is model-agnostic and has been validated on diverse architectures (Two-stage, Single-stage, and Transformer-based detectors like RF-DETR).

In conclusion, ASD represents a significant advancement in adversarial defense by shifting the focus from pixel-space manipulation to spectral-domain analysis, effectively neutralizing the unique characteristics of physically realizable adversarial attacks.