Technical Summary: Towards Universal Physical Adversarial Attacks via a Joint Multi-Objective and Multi-Model Optimization Framework

1. Problem Statement

Physical adversarial attacks, particularly 3D adversarial camouflage, face three critical bottlenecks that limit their real-world deployment against black-box models:

1. Overfitting to Single Surrogates: Existing methods often rely on a single surrogate model (e.g., YOLO series), causing generated camouflage to overfit specific decision boundaries and fail against heterogeneous networks.
2. Lack of Synergy Between Shallow and Deep Attacks: Current strategies typically employ either shallow output suppression (efficient but prone to overfitting) or deep feature disruption (transferable but lacking rigid output guidance). The absence of a synergistic integration prevents attacks from achieving both high efficiency and deep generalization.
3. Gradient Conflicts in Multi-Source Fusion: When integrating multiple models and objectives, naive fusion strategies (e.g., equal-weight summation) lead to severe gradient conflicts and oscillations within the restricted parameter space of physical textures. This results in directional cancellation and energy loss, masking the adversarial demands of robust models.

Furthermore, existing methods struggle to generalize across different visual tasks (e.g., transferring from object detection to semantic segmentation or depth estimation) due to the semantic gaps in underlying optimization objectives.

2. Methodology: Joint Multi-Objective and Multi-Model Optimization Framework (JMOF)

The proposed JMOF framework addresses these limitations through a four-pronged approach:

A. Quantitative Surrogate Model Selection

Instead of intuitive or random selection, JMOF utilizes Gradient Cosine Similarity to quantify the heterogeneity of candidate models. A Greedy Selection algorithm constructs an ensemble of $N$ models that minimizes pairwise gradient similarity. This ensures the ensemble covers a diverse spectrum of visual cognitive logic, preventing the optimization from converging into local minima shared by similar architectures.

B. Dual-Level Multi-Objective Attack Mechanism

The framework employs a joint loss function comprising three components to balance efficiency and generalization:

1. Detection Loss ($L_{det}$): A LogSumExp-based suppression strategy that indiscriminately suppresses the confidence scores of all predicted bounding boxes with high Intersection over Union (IoU) with the ground truth. This provides explicit, shallow guidance.
2. Feature Loss ($L_{fea}$): A deep disruption mechanism that targets intermediate feature maps. It performs Mean Suppression (attenuating overall activation) and Variance Flattening (smoothing semantic peaks) across multi-scale layers. This dismantles shared spatial structural features to enhance black-box transferability.
3. Smooth Loss ($L_{smo}$): A Total Variation-based loss ensuring the continuity of the adversarial texture, eliminating high-frequency noise that causes failure under physical viewpoint variations and occlusions.

C. Orthogonal Gradient Alignment (OGA)

To resolve gradient conflicts, JMOF introduces the OGA strategy. Unlike passive discarding methods (e.g., DRF, DGR), OGA actively restructures conflicting gradients:

• Local Patch Analysis: The global texture is divided into local patches. For each patch, a Gram matrix is computed to capture geometric angles and magnitude correlations among heterogeneous gradients.
• Orthogonal Projection: Through eigenvalue decomposition, OGA constructs an orthogonal transformation matrix. This projects conflicting gradients onto an orthogonal basis, effectively converting repulsive optimization potential energy into synergistic kinetic energy without discarding feature preferences.
• Dynamic Task Weighting: A dynamic weighting mechanism ($\omega$) assigns higher weights to models with higher adversarial difficulty (e.g., Vision Transformers), ensuring the attack evolves toward high-dimensional vulnerability spaces.
• Adaptive Step-Size: The orthogonal decomposition naturally compresses strongly repulsive components, acting as an adaptive step-size decay to prevent optimizer stagnation in steep loss regions.

D. Enhancement Strategies

• View-Dependent Texture Gater (VTG): Enforces visibility constraints during backpropagation, ensuring that invisible pixels (occluded by self-occlusion or viewpoint shifts) are not erroneously smoothed away, preserving adversarial energy on visible regions.
• Stochastic Texture Dropout (STD): Randomly masks blocks of the adversarial texture during training to simulate real-world occlusions. This forces the algorithm to avoid reliance on specific local shortcuts and learn globally generalizable features.

3. Key Contributions

1. Gradient Similarity Analysis: The paper reveals a critical trade-off in gradient similarity: overly aligned gradients limit diversity, while highly divergent gradients cause conflicts. It defines an effective similarity regime to guide the selection of optimal surrogate ensembles.
2. Dual-Level Attack Mechanism: The proposal of a synergistic mechanism that jointly suppresses prediction outputs and flattens intermediate feature distributions, overcoming the limitations of single-level attacks.
3. Orthogonal Gradient Alignment (OGA): A novel module that resolves cross-model gradient conflicts by transforming mutually repulsive gradients into synergistic optimization directions, enabling convergence on shared vulnerable regions across heterogeneous models.
4. Cross-Task Generalization: The demonstration that the framework can generate attacks capable of simultaneously deceiving object detection, semantic segmentation, and monocular depth estimation models, bridging the gap between high-level semantic perception and low-level geometric regression.

4. Experimental Results

Extensive experiments were conducted using the CARLA simulator and physical 1:18 scale models with PET film carriers.

• Cross-Model Transferability: JMOF outperformed state-of-the-art baselines (e.g., FCA, DAS, HTCA, TACO) across diverse black-box detectors, including CNNs (YOLO, Faster R-CNN), Vision Transformers (ViT, Swin), and hybrids. It maintained high Attack Success Rates (ASR) even against highly heterogeneous models where single-model baselines failed.
• Multi-View Robustness: The method demonstrated superior robustness across varying azimuth and pitch angles, particularly on side views where traditional methods degrade significantly due to topological changes.
• Ablation Studies:
  • Removing the feature loss ($L_{fea}$) caused a >13% drop in ASR against CNNs and >15% against ViTs, confirming the necessity of deep feature disruption.
  • Removing OGA and using naive summation resulted in severe performance degradation, validating the need for active gradient restructuring.
  • The inclusion of VTG and STD modules improved ASR by approximately 9% and 6%, respectively.
• Cross-Task Performance: When jointly optimizing for object detection and semantic segmentation (or depth estimation), JMOF with OGA successfully disrupted both tasks simultaneously. In contrast, naive summation strategies failed to maintain performance on both tasks due to gradient conflicts.
• Physical Deployment: In real-world physical experiments, the method achieved ASRs of 65.71% (SMs), 72.76% (CNNs), and 70.63% (ViTs), representing a robust performance despite the domain gap from simulation.

5. Significance and Claims

The paper claims to advance the generalization limits of physical adversarial attacks by providing a robust framework for evaluating visual AI vulnerabilities in real-world deployments.

• Universal Applicability: The framework is presented as a universal underlying logic capable of quantifying and resolving gradient conflicts across different architectures and tasks. It eliminates the need for blind trial-and-error when integrating new models; instead, it uses gradient similarity analysis to adaptively incorporate new architectures into the joint optimization.
• Bridging the Gap: The research addresses the massive gap between the capability boundaries of current physical attacks and the demands of real-world black-box, multi-task scenarios.
• Synergistic Optimization: By transforming conflicting gradients into synergistic directions, the method achieves a paradigm shift from passive evasion to active restructuring, enabling physical adversarial camouflage to converge on shared vulnerability spaces across diverse networks and visual tasks.

The authors acknowledge that while the method achieves breakthrough performance, it entails increased computational costs due to multi-model forward and backward propagation. Future work is proposed to explore more efficient joint-optimization algorithms and lightweight mechanisms to balance generation efficiency with generalization performance.