Digital-to-Physical Transfer of Adversarial Patches for Aerial Vehicle Detection

Technical Summary: Digital-to-Physical Transfer of Adversarial Patches for Aerial Vehicle Detection

Problem Statement
Deep neural network (DNN)-based object detectors, such as YOLOv3, are widely deployed for analyzing aerial and satellite imagery in applications ranging from environmental monitoring to urban analytics. While these models demonstrate high accuracy, they are known to be vulnerable to adversarial examples. Although physical adversarial attacks using printable patterns have been studied extensively in ground-based scenarios (e.g., autonomous driving), their effectiveness in aerial contexts remains underexplored. Aerial imagery presents unique challenges, including large observation distances, varying viewing angles, significant scale variations, and environmental degradation caused by weather (fog, haze, illumination changes). This study addresses the critical gap in understanding whether adversarial patches optimized in the digital domain can effectively transfer to the physical domain to degrade aerial vehicle detection performance under real-world capture conditions.

Methodology
The authors propose a digital-to-physical evaluation pipeline involving three main components:

1. Patch Configurations: Three deployment strategies are evaluated:

   • ON Patch: Placed directly on the vehicle roof, modifying the object's visual appearance.
   • OFF Patch: Placed around the vehicle in a $\Pi$-shaped structure, manipulating contextual information.
   • OFF-Side Patch: Two patches placed on the left and right sides of the vehicle, manipulating a subset of the context.

2. Optimization Process: Adversarial patches are optimized in the digital domain using a YOLOv3 detector fine-tuned on the Cars Overhead With Context (COWC) dataset. The optimization objective minimizes the maximum objectness score of the target vehicle while incorporating constraints to ensure physical realizability:

   • Loss Function: $L_i(P) = \max(\hat{S}^u_i) + \delta \cdot \text{NPS}(P) + \gamma \cdot \text{TV}(P)$
   • Non-Printability Score (NPS): Penalizes colors that cannot be reliably reproduced by physical printers.
   • Total Variation (TV): Encourages spatial smoothness to reduce abrupt color changes.
   • Augmentation: The training pipeline incorporates geometric transformations (scaling, rotation), color-space adjustments (brightness, contrast), and weather-based augmentations (snow, rain, fog, etc.) to simulate environmental variability.

3. Evaluation Metrics:

   • Digital Domain: Measured using the Average Objectness Reduction Rate (AORR) and mean Average Precision (mAP) on the Side Street dataset.
   • Physical Domain: Measured using the Objectness Score Ratio (OSR) on real-world aerial footage captured from a 40m building (Side Street) and a UAV at 60m (Car Park). OSR compares the average objectness score of frames with the patch against frames without it.

Key Results
The experimental results reveal a significant discrepancy between digital simulations and physical deployments:

• Digital vs. Physical Effectiveness: In the digital domain, the OFF patch (contextual manipulation) achieved the highest attack effectiveness, with an AORR of 85.51% under standard testing. However, in the physical domain, the ON patch (object-level manipulation) demonstrated superior robustness, achieving Objectness Score Ratios (OSR) between 0.197 and 0.343. The OFF patch performed less effectively physically (OSR up to 0.748), likely due to instability caused by viewpoint changes, occlusion, and misalignment in real-world conditions.
• Impact of Weather Augmentation: Contrary to the intuition that simulating weather improves robustness, the study found that weather-based augmentation did not improve attack performance. The training pipeline without weather augmentation (G/C) consistently outperformed the pipeline with weather augmentation (G/C+W) in both digital and physical evaluations. The authors suggest that excessive weather variability may hinder the optimization of discriminative adversarial patterns.
• Role of TV Regularization: The Total Variation coefficient ($\gamma$) significantly impacts performance. Moderate regularization (e.g., $\gamma = 2.5$) provided the best trade-off between printability and attack potency. Excessive smoothing ($\gamma = 40.0$) degraded the attack by removing necessary high-frequency patterns, while insufficient smoothing resulted in less effective patches.

Contributions and Significance
The paper makes the following contributions to the field of aerial object detection security:

1. Digital-to-Physical Evaluation: It presents a comprehensive evaluation of adversarial patch attacks specifically for aerial vehicle detection, highlighting the practical transfer gap between digital optimization and real-world aerial capture.
2. Strategy Comparison: It compares object-level (ON) and context-level (OFF, OFF-Side) deployment strategies, demonstrating that the most effective strategy differs fundamentally between digital and physical domains. Contextual attacks are potent in simulation but fragile in reality, whereas direct object attacks are more robust in physical environments.
3. Empirical Evidence on Augmentation: It provides empirical evidence that weather-based augmentation does not necessarily improve, and may even hinder, adversarial patch optimization in the aerial domain.
4. Security Implications: The findings underscore that aerial object detection systems are vulnerable to physically deployable adversarial patches. The study emphasizes that robustness assessments for aerial detectors cannot rely solely on digital experiments; they must account for physical deployment conditions, environmental variability, and the specific stability of object-level versus contextual perturbations.

The authors conclude that while digitally optimized patches can transfer to real-world aerial environments and significantly degrade detector performance, the optimal attack configuration is domain-dependent. These insights are critical for understanding the practical vulnerabilities of aerial perception systems and for developing more robust defense mechanisms.