Technical Summary: AdvScene

Problem Statement

Adversarial patches are physical patterns attached to real objects designed to mislead AI vision systems. While existing research has demonstrated that these patches can succeed in specific, controlled scenarios, current evaluation methods fail to adequately measure scene robustness: the effectiveness of a fixed deployed patch across the changing conditions of a real environment (e.g., varying viewpoints, distances, and scene contexts).

The paper identifies a critical gap in existing evaluation paradigms:

1. Image-plane transformations (e.g., EOT-style sweeps) treat patches as 2D overlays, failing to account for physical projection effects like oblique angles, partial occlusion, or scale changes that occur when a patch is attached to a 3D surface.
2. Real-capture benchmarks provide physical evidence but are limited to fixed poses and conditions, making it labor-intensive and often unsafe to densely sample the operational space required to understand deployment risk.
3. Simulator-based platforms offer controllability but often lack grounding in specific real-world scenes, potentially introducing artifacts that do not reflect actual deployment sites.

Consequently, a patch that succeeds in a single "anchor view" may pose limited practical risk, while the true threat lies in whether the attack remains effective across a broad range of naturally occurring views. The central problem is how to measure the operational envelope—the specific region of scene-induced observation conditions where a fixed deployed attack succeeds—without requiring exhaustive physical experimentation.

Methodology: AdvScene Framework

The authors propose AdvScene, a scene-grounded framework that reconstructs real environments and evaluates fixed adversarial patches within them. The core innovation is reframing evaluation as an operational measurement problem rather than a single-image success rate.

1. Scene Reconstruction and Embedding

AdvScene utilizes 3D Gaussian Splatting (3DGS) to create high-fidelity, controllable digital twins of real scenes from multiview captures. The framework follows a four-stage pipeline:

• Scene Reconstruction: A clean 3DGS scene ($G_0$) is reconstructed from multiview images of a real scene.
• Anchor Attack Instantiation: An external attack pipeline generates a fixed adversarial patch from a selected anchor view, producing an attacked image ($I^*_a$) and a patch mask ($M_a$).
• Adversarial Patch-to-Scene Embedding (APSE): This is the core technical contribution. APSE lifts the 2D anchor-view attack into the 3D scene to create an adversarial scene ($G_{adv}$). It addresses the "constrained lifting problem": how to represent a patch defined in one view such that it remains faithful, localized, and surface-attached under novel viewpoints.
• Scene-Level Evaluation: The adversarial scene is rendered under controlled variations in camera pose, distance, and scene context to estimate the operational envelope.

2. Adversarial Patch-to-Scene Embedding (APSE)

APSE resolves cross-view ambiguity while preserving attack-critical appearance. It employs a hybrid Gaussian representation consisting of:

• Reused Gaussians ($O$): Existing scene Gaussians within the patch region are reused to maintain the reconstructed surface geometry. Only their appearance parameters (color, opacity) are updated.
• Spawned Gaussians ($N$): New Gaussians are densely spawned within the patch mask to provide the local capacity needed for high-frequency adversarial patterns.

To ensure the patch behaves physically correctly, APSE optimizes under four constraints:

1. Anchor-View Fidelity: The rendered patch must match the attacked anchor image ($L_{patch}$).
2. Scene Preservation: Regions outside the patch must remain identical to the clean reconstruction ($L_{keep}$).
3. Auxiliary-View Containment: Using unattacked auxiliary views, the method suppresses "leakage" (unintended appearance changes in the background) and floating artifacts ($L_{aux-rgb}$, $L_{aux-alpha}$).
4. Surface Attachment: To prevent the patch from floating or detaching (especially on thin objects like traffic signs), APSE initializes spawned Gaussians on a fitted foreground plane and constrains their movement during optimization using local surface frames and normal-attachment losses ($L_{normal}$).

3. Operational Envelope Measurement

The output of AdvScene is not a single metric but an operational envelope ($E$), defined as the set of observation conditions $\theta$ (parameterized by camera orientation, position relative to the object, and intrinsics) where the attack succeeds ($S(\theta) = 1$). This allows for systematic analysis of where and why an attack fails as scene conditions change.

Key Contributions

1. Conceptual Shift: The paper identifies scene robustness as a missing dimension in physical adversarial patch evaluation and formulates the problem as measuring the operational envelope of a fixed patch in reconstructed real scenes.
2. AdvScene Framework: A scene-grounded evaluation system combining real-scene reconstruction, fixed-patch embedding, and controlled novel-view rendering to measure attack behavior under realistic deployment variations.
3. APSE Method: A constrained embedding technique for 3D Gaussian Splatting that preserves attack-critical appearance while enforcing locality, clean-scene preservation, target-surface attachment, and cross-view stability.
4. Empirical Validation:
   • Fidelity: Validated against 7 real-world physical scenes, AdvScene achieves 99.30% consistency in Attack Success Rate (ASR) with physical captures, significantly outperforming image-plane transformation baselines (93.54%). Crucially, it avoids the baseline's severe overestimation of risk in close, oblique views.
   • Scale: Evaluated across 59 CO3D scenes (object-centered) and 28 Waymo scenes (driving scenarios) covering targeted/untargeted classification and object detection disappearance.
   • Findings:
     • Anchor-view success often collapses outside a narrow operational envelope; a patch succeeding in one view does not guarantee broad deployment robustness.
     • Driving scenes induce directional and asymmetric operational envelopes due to vehicle trajectories and roadside geometry.
     • Geometry-aware EOT (Expectation over Transformation) is the primary technique that expands the operational envelope. Regularizers like Non-Printability Score (NPS) and Total Variation (TV) mainly affect printability or smoothness but do not fundamentally alter the scene-level failure modes.
     • $\ell_2$ regularization can constrain patch magnitude without substantially shrinking the operational envelope, suggesting visual stealth and scene-level robustness are not always in direct conflict.

Significance and Claims

The paper claims that physical patch risk should be understood as an envelope rather than a point estimate. By making this envelope measurable, AdvScene provides a more faithful basis for evaluating physical adversarial risk.

The authors emphasize that AdvScene is not a stronger attack generator but a measurement framework for fixed attacks produced by existing pipelines. Its significance lies in revealing systematic failure modes (e.g., oblique view collapse) that image-centric or simulator-based evaluations miss. This allows defenders to understand the specific operational conditions under which a deployed patch remains a threat and helps attack designers understand which techniques (specifically geometry-aware EOT) actually improve scene-level robustness versus those that only improve physical plausibility or anchor-view performance.

The work is modest in scope, focusing on fixed, printable, surface-attached 2D patches in static scenes, explicitly excluding 3D adversarial objects, dynamic displays, or full autonomous driving stacks with sensor fusion, leaving those for future work.