PASTA: A Patch-Agnostic Twofold-Stealthy Backdoor Attack on Vision Transformers

1. Problem Statement

Vision Transformers (ViTs) have become dominant in computer vision but are vulnerable to backdoor attacks. Existing patch-wise backdoor attacks against ViTs suffer from two critical limitations:

1. Location Sensitivity: Traditional attacks rely on a pre-defined trigger insertion location during training. If the trigger is activated at a different location during inference, the attack often fails.
2. Lack of Stealthiness: Existing ViT-specific attacks (e.g., BadViT, TrojViT) maximize model attention on the trigger to ensure effectiveness. This results in obvious visual artifacts and abnormal attention maps, making them easily detectable by both human inspectors and machine-based defenses (e.g., attention analysis).

The authors pose three key research questions:

• Is attack effectiveness sensitive to trigger activation locations (TALs) in ViTs compared to CNNs?
• Can training with triggers in multiple locations synergistically enhance the "Trigger Radiating Effect" (TRE)?
• How can an attack achieve high effectiveness across arbitrary patches while maintaining both visual and attention stealthiness?

2. Key Observations

Before proposing their solution, the authors conducted empirical studies revealing:

• Trigger Radiating Effect (TRE): Unlike CNNs, where attack effectiveness drops drastically if the trigger moves from its training location, ViTs exhibit a strong "Trigger Radiating Effect." A patch-wise trigger inserted during training remains highly effective even when activated in neighboring patches during inference. This is attributed to the self-attention mechanism, which captures long-range dependencies across patches.
• Perturbation Magnitude: Larger trigger perturbations yield stronger TRE but compromise visual stealthiness.
• Synergistic Training: Training a backdoor model by inserting triggers into multiple patch locations (inter-patch manner) synergistically enhances the TRE compared to single-patch insertion.

3. Methodology: PASTA

The authors propose PASTA (Patch-Agnostic Twofold-Stealthy Attack), a framework designed to achieve high attack success rates across arbitrary patches while evading detection.

A. Multi-Location Trigger Insertion Strategy (MIS)

To maximize TRE, PASTA does not fix the trigger to a single location. Instead, it defines a candidate set of patch locations (including central patches of quadrants and corner patches). For each poisoned sample, a specific patch location is sampled from this set using a hierarchical sampler. This ensures the model learns to associate the trigger pattern with the target class regardless of where the trigger appears, thereby achieving "Patch-Agnostic" activation.

B. Twofold Stealthiness

PASTA targets two dimensions of stealth:

1. Visual Stealthiness: Minimizes the pixel-level difference ($l_2$-norm) between clean and poisoned images to avoid human detection.
2. Attention Stealthiness: Minimizes the disparity between the attention maps of clean and poisoned samples. Unlike prior attacks that maximize attention on the trigger, PASTA explicitly minimizes the difference in attention distribution to evade machine inspection.

C. Adaptive Bi-Level Optimization Framework

The core challenge is that optimizing the trigger ($t$) and the model parameters ($\theta$) simultaneously is difficult because they are non-separable in the loss function; changes in one immediately affect the loss landscape of the other, leading to local optima.

PASTA formulates the attack as a Bi-Level Optimization Problem:

• Upper Level: Optimize model parameters $\theta$ to minimize clean task loss and backdoor loss (given the optimal trigger).
• Lower Level: Optimize the trigger $t$ to minimize backdoor loss, attention disparity, and visual disparity (given the current model).

To solve this, they introduce an Adaptive Backdoor Training Framework:

• Instead of fully optimizing one before the other, the framework alternates between updating the trigger and the model parameters in small, incremental steps (small epochs).
• This allows $t$ and $\theta$ to gradually adapt to each other's updates, preventing the system from converging to sub-optimal local minima and ensuring the trigger remains effective and stealthy as the model evolves.

4. Key Contributions

1. Discovery of TRE: First to observe and quantify the "Trigger Radiating Effect" in ViTs, proving that self-attention allows triggers to activate backdoors across neighboring patches, a phenomenon absent in CNNs.
2. New Attack Payload: Proposes a novel payload where the backdoor can be activated by a trigger in arbitrary patches during inference, bypassing the need for fixed trigger locations.
3. Twofold Stealthiness: Achieves simultaneous visual and attention imperceptibility, a significant improvement over existing ViT attacks that sacrifice stealth for effectiveness.
4. Adaptive Optimization: Introduces a bi-level optimization framework with an adaptive training loop to effectively solve the non-separable optimization problem of joint trigger and model learning.

5. Experimental Results

The authors evaluated PASTA on four datasets (CIFAR-10, CIFAR-100, ImageNet, Sub-ImageNet) against state-of-the-art (SOTA) attacks and defenses.

• Attack Effectiveness:

  • Achieved an average Attack Success Rate (ASR) of 99.13% across arbitrary patches (TRE), significantly outperforming existing patch-wise attacks (e.g., BadViT, DBIA) which dropped to ~45% when tested on random patches.
  • Maintained high Clean Accuracy (ACC), with drops limited to ~1% on average.

• Stealthiness:

  • Visual: Improved visual stealthiness by 144.43× (measured by $l_2$-norm) compared to SOTA attacks.
  • Attention: Improved attention imperceptibility by 18.68× (measured by $l_2$-norm of attention map differences).

• Robustness against Defenses:

  • Patch Operations: While single-location attacks failed against "Patch Drop" and "Patch Shuffle" defenses, PASTA's multi-location strategy maintained high ASR (up to 95.2%).
  • Attention Defenses: PASTA evaded attention-based detection (DBAVT) with a False Negative Rate (FNR) of nearly 0% for poisoned samples (meaning it was rarely detected), whereas other attacks were detected more frequently.
  • Test-Time Blocking: Against BAVT (which blocks the highest attention patch), PASTA achieved 100% ASR when using multiple trigger locations, rendering the defense ineffective.

6. Significance

This paper fundamentally shifts the understanding of backdoor attacks on Vision Transformers. It demonstrates that the self-attention mechanism, often touted as a robust feature, actually creates a unique vulnerability (TRE) that allows for patch-agnostic attacks. By combining this insight with a sophisticated bi-level optimization framework, PASTA sets a new benchmark for stealth and robustness, highlighting that current defenses focusing on fixed trigger locations or single-point attention anomalies are insufficient against modern ViT architectures. The work underscores the urgent need for new defense mechanisms that account for the global, long-range nature of ViT attention.