ColorFD: A Finite-Difference Guided Black-Box Physical Adversarial Attack for Remote Sensing Object Detection

Technical Summary: ColorFD: A Finite-Difference Guided Black-Box Physical Adversarial Attack for Remote Sensing Object Detection

Problem Statement
Deep neural network (DNN)-based remote sensing object detectors, while effective, remain vulnerable to adversarial perturbations. Existing research predominantly focuses on digital or white-box settings. Black-box physical attacks, which are more representative of real-world deployment constraints, remain underexplored. These attacks face significant challenges: ensuring physical feasibility (e.g., printability and concealment) and overcoming inefficient optimization within high-dimensional search spaces when model gradients are unavailable. Furthermore, existing physical attack methods are often evaluated primarily in the digital domain, lacking sufficient validation in real physical environments, particularly for remote sensing scenarios characterized by substantial scale variations and complex imaging conditions.

Methodology
The paper proposes ColorFD, a black-box physical adversarial attack framework designed for remote sensing object detectors. The method models adversarial perturbations as multiple small, pure-color square patches rather than pixel-wise texture optimizations, facilitating fabrication and physical deployment. The core methodology integrates three primary components:

1. Differential Evolution (DE) Optimization: The positions and color parameters of the patches are jointly optimized using DE, a population-based algorithm that does not require gradient information. The attack objective is formulated as minimizing the detector's response to induce object disappearance.
2. Target-wise Fitness and Selection Mechanism: To address the complexity of multi-target attacks where a single global fitness metric may obscure individual target states, ColorFD introduces a target-wise evaluation. A target-level fitness function evaluates the attack state of each target independently. A novel selection mechanism preserves target-specific advantageous variables during the evolutionary process by performing partial component exchanges between parent and offspring, thereby alleviating optimization conflicts across multiple targets.
3. Search Space Reduction via Guidance Strategies: To mitigate the computational cost of high-dimensional search spaces, two guidance strategies are employed:
   • Key-Region Localization: Utilizing finite-difference color probing, this strategy iteratively identifies instance-specific sensitive regions within the target bounding box. By applying sliding windows and multi-color perturbations, it generates response heatmaps to constrain the search space to high-impact areas.
   • Common-Feature Extraction: To further reduce redundant computations, this strategy aggregates key-region distributions across instances of the same category. By aligning targets and generating a category-level common-feature heatmap, it provides spatial priors that guide the search without requiring repeated per-instance localization.

Key Contributions
The authors summarize their contributions as follows:

1. Physically Realizable Design: A proposal for an adversarial design based on multiple small pure-color square patches, which balances strong attack effectiveness with ease of fabrication and deployment in real-world environments.
2. Target-wise Optimization Mechanism: The development of a target-wise fitness and selection mechanism for multi-target black-box attacks. This approach evaluates targets separately and preserves target-level improvements, addressing the limitations of single global fitness metrics in multi-target scenarios.
3. Guidance Strategies: The introduction of two strategies to constrain the patch search space: a key-region strategy using finite-difference probing for instance-specific sensitivity, and a common-feature strategy that constructs category-level priors to reduce repeated localization while maintaining competitive performance.

Experimental Results
Experiments were conducted on the DIOR remote sensing dataset, specifically utilizing an aircraft subset containing 407 instances across various types, scales, and orientations. The method was evaluated against YOLOv3u, YOLOv5u, and Faster R-CNN.

• Digital Domain Performance: ColorFD outperformed the tested black-box patch method (Bbox-Att) across all evaluated detectors. On YOLOv5u, the key-region guided variant achieved an Attack Success Rate (ASR) of 79.61% and reduced AP50 to 0.1453, surpassing the black-box baseline and remaining competitive with strong white-box baselines (AP-PA and BADEI). On Faster R-CNN, the unguided version achieved an ASR of 95.09% with an AP50 of 0.0169.
• Guidance Strategy Efficacy: Key-region guidance significantly improved performance on one-stage detectors (YOLOv3u, YOLOv5u) but showed limited gains on the two-stage Faster R-CNN, likely due to the inaccessibility of intermediate Region Proposal Network (RPN) information in black-box settings. Conversely, common-feature guidance demonstrated superior performance on Faster R-CNN (ASR of 99.35%), effectively compensating for the limitations of key-region localization on two-stage detectors.
• Physical World Validation: Physical experiments were conducted using colored paper patches on alloy aircraft models under controlled imaging conditions. Despite minor deviations in color and position due to material properties and manual attachment, the optimized patches successfully suppressed detections in the physical domain, demonstrating digital-to-physical transferability.

Significance and Claims
The paper claims that ColorFD provides a practical robustness-evaluation approach for optical remote sensing detection systems under restricted model access. It demonstrates that black-box physical attacks can achieve performance levels comparable to, or exceeding, state-of-the-art white-box attacks when appropriate optimization strategies and guidance mechanisms are employed. The study highlights that the proposed pure-color patch design, combined with target-wise optimization and finite-difference guided constraints, effectively balances attack effectiveness with physical feasibility.

The authors acknowledge limitations, including the stochastic nature of DE-based optimization leading to performance variability, the reduced efficacy of key-region localization on two-stage detectors due to lack of intermediate information access, and the sensitivity of pure-color patches to environmental variations such as viewpoint and scale shifts. Future work is identified as focusing on improving optimization stability and the visual stealthiness of the patches.