💻 computer science

STRAP-ViT: Segregated Tokens with Randomized -- Transformations for Defense against Adversarial Patches in ViTs

STRAP-ViT is a non-trainable, plug-and-play defense mechanism for Vision Transformers that mitigates adversarial patch attacks by statistically identifying anomalous tokens via Jensen-Shannon Divergence and applying randomized transformations to them, achieving near-clean accuracy without additional training costs.

Original authors: Nandish Chattopadhyay, Anadi Goyal, Chandan Karfa, Anupam Chattopadhyay

Published 2026-03-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nandish Chattopadhyay, Anadi Goyal, Chandan Karfa, Anupam Chattopadhyay

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, high-tech security guard (a Vision Transformer or ViT) whose job is to look at photos and identify what's in them. This guard is incredibly powerful; it doesn't just look at pixels like a human eye, it breaks the image into tiny puzzle pieces called "tokens" and analyzes how they all relate to each other to make a decision.

However, there is a clever trickster known as an Adversarial Patch. Think of this as a tiny, brightly colored sticker or a weirdly patterned piece of tape that a criminal sticks onto a stop sign or a person's shirt. To a human, it's just a small sticker. But to the AI guard, this sticker is like a hypnotic lighthouse. It screams so loudly that the guard ignores the actual stop sign or person and focuses entirely on the sticker, confidently misidentifying the object (e.g., thinking a stop sign is a "radiator").

This paper introduces a new defense system called STRAP-ViT. Here is how it works, explained simply:

1. The Problem: The "Loud" Sticker

The authors realized that the puzzle pieces (tokens) covering the malicious sticker behave differently than the normal pieces.

Normal tokens are calm and follow the usual patterns of the image.
Sticker tokens are chaotic, noisy, and statistically "weird." They have a different "vibe" or energy distribution.

2. The Solution: STRAP-ViT (The Smart Filter)

STRAP-ViT is like a bouncer standing at the door of the AI's brain. It doesn't need to be retrained; it just plugs in and starts working. It operates in two stages:

Stage A: The Detective (Detection)

The bouncer uses a mathematical tool called Jensen-Shannon Divergence (let's call it the "Weirdness Meter").

It compares every single puzzle piece in the image against what a "normal" piece should look like.
If a piece is too "weird" (high divergence), the bouncer flags it.
The Rule: The bouncer doesn't need to catch every weird piece. It only needs to catch enough to cover at least 50% of the sticker. It's like realizing that if you cover half of a hypnotic lighthouse with a blanket, the light isn't strong enough to blind the guard anymore.

Stage B: The Magic Wiper (Mitigation)

Once the bouncer identifies the suspicious tokens (the ones under the sticker), it doesn't just delete them (which would leave a hole in the picture). Instead, it applies a randomized "magic wiper" to them.

Imagine the tokens are covered in sticky, confusing glue (the adversarial noise).
The system randomly chooses a method to scrub that glue off: maybe it stretches the token, squishes it, or changes its color temperature.
Why Random? If the trickster tries to make a sticker that survives one type of scrubbing, the system just picks a different random scrubbing method next time. It's impossible for the attacker to predict which random move will happen, so the sticker's power is neutralized.

3. The Result: A Safe Image

After the bouncer scrubs the suspicious tokens, the AI guard looks at the image again.

The "hypnotic lighthouse" is now dim and broken.
The guard can finally see the real object (the stop sign or the monkey) and classify it correctly.
Crucially, because the system only touches the "weird" tokens and leaves the rest of the image alone, the AI's performance on normal, clean photos barely changes. It's like fixing a scratch on a car without repainting the whole thing.

Why This Matters

No Heavy Lifting: Unlike other defenses that require retraining the AI (which costs millions of dollars and takes years), STRAP-ViT is a "plug-and-play" tool. You just add it to the existing system.
Universal: It works on different types of AI guards and different types of stickers, whether the sticker is big or small, or if there are multiple stickers on one image.
Efficient: It's so fast and lightweight that it doesn't slow down the AI, making it perfect for real-world use like self-driving cars or security cameras.

In a nutshell: STRAP-ViT is a smart, lightweight filter that spots the "loud" noise of a malicious sticker, randomly scrambles that noise until it's harmless, and lets the AI see the truth again—all without needing a massive software update.

1. Problem Statement

Vision Transformers (ViTs) are increasingly central to AI but remain highly vulnerable to adversarial patches. Unlike pixel-level perturbations, adversarial patches are localized, physically realizable patterns (e.g., stickers) that can hijack the model's self-attention mechanism.

The Threat: A small, high-contrast patch can dominate the attention weights across layers, corrupting the "class token" and forcing confident misclassifications, even if the patch covers a tiny fraction of the image.
Limitations of Existing Defenses:
- Certifiable methods (e.g., smoothed ViTs) are computationally expensive and often require specialized architectures.
- Empirical methods (e.g., adversarial training) are heavy on compute and lack transferability.
- Pixel-space defenses often fail to address the specific token-level reasoning mechanisms of ViTs.
The Gap: There is a need for a defense mechanism that is training-free, computationally efficient, model-agnostic, and operates directly within the ViT's semantic feature space (tokens) rather than the raw pixel space.

2. Methodology: STRAP-ViT

STRAP-ViT is a non-trainable, plug-and-play defense block inserted into the ViT inference pipeline. It operates in two distinct phases: Detection and Mitigation.

A. Detection Phase: Token Segregation via Jensen-Shannon Divergence (JSD)

The core insight is that tokens overlapping with an adversarial patch exhibit distinct statistical properties compared to clean tokens.

Statistical Basis: Adversarial patches cause a systematic shift in the channel distribution of affected tokens, leading to higher entropy and statistical divergence from a "clean" reference distribution.
Metric: The method uses Jensen-Shannon Divergence (JSD) to measure the discrepancy between a token's probability distribution ( $p_{\ell,t}$ $p_{ℓ, t}$ ) and a pre-computed reference distribution ( $q_{\ell,t}$ $q_{ℓ, t}$ ) derived from clean images.
- $JSD(p|q) = \frac{1}{2}KL(p||M) + \frac{1}{2}KL(q||M)$ , where $M$ is the midpoint distribution.
- The anomaly score is defined as $s_{\ell,t} = \sqrt{JSD(p_{\ell,t}|q_{\ell,t})}$ .
Selection: Tokens with the highest JSD scores are flagged as anomalous. The system selects the top $K$ tokens (a hyperparameter) for transformation. The value of $K$ is chosen to ensure at least 50% of the adversarial patch area is covered by the transformed tokens.

B. Mitigation Phase: Randomized Composite Transformations

Once anomalous tokens are identified, they undergo a randomized composite transformation to neutralize the adversarial noise without retraining the model.

Randomization Strategy: Instead of applying a single fixed transformation, the system randomly selects a subset and an order of transformations for each token. This prevents attackers from optimizing patches against a static defense (robust against Expectation Over Transformation attacks).
Transformation Types:
1. $L_p$ -Norm Projection: Projects the token embedding onto a ball of radius $r$ to curb extreme channel energy.
2. Affine Contraction: Applies a bounded linear transformation ($Az + b$) where the operator norm is $<1$ , effectively shrinking the feature space.
3. Simplex Transform (Temperature Scaling): Applies a softmax with a temperature parameter $T$ to re-weight channels and dampen spikes.
Process: The transformed tokens are re-integrated into the sequence, and the ViT proceeds with the standard forward pass to generate the final prediction.

3. Key Contributions

Novel Defense Mechanism: Introduces STRAP-ViT, a training-free method that exploits the statistical divergence of adversarial tokens to detect and surgically neutralize them.
Token-Level Operation: Unlike pixel-space defenses, STRAP-ViT operates directly on the semantic token representations used by the ViT for reasoning, making it inherently suitable for Transformer architectures.
Model and Patch Agnostic: The method does not require retraining the backbone model and works across different ViT architectures (e.g., ViT-base-16, DinoV2) and various patch types.
Randomized Robustness: The use of randomized composite transformations ensures that the defense is robust against adaptive attacks that attempt to bypass a specific transformation.

4. Experimental Results

The authors evaluated STRAP-ViT on ImageNet and CalTech-101 using ViT-base-16 and DinoV2 backbones against multiple attacks: Adversarial Patch, LAVAN, GDPA, and RP2 (multiple patches).

Robust Accuracy:
- STRAP-ViT achieved robust accuracies within a 2–3% range of the clean baselines.
- Example (ViT-base-16 on ImageNet):
  - Adversarial Patch (40x40): Clean baseline ~80.5% $\rightarrow$ No Defense ~3.6% $\rightarrow$ STRAP-ViT 78.8%.
  - LAVAN (40x40): No Defense ~7.4% $\rightarrow$ STRAP-ViT 79.2%.
  - GDPA (40x40): No Defense ~31.9% $\rightarrow$ STRAP-ViT 77.2%.
Clean Accuracy Impact: The defense has minimal impact on clean data. On ImageNet, the drop in clean accuracy was negligible (e.g., -0.4%), proving the method does not degrade performance on benign inputs.
Multiple Patches: The method successfully defended against RP2, an attack using up to five distributed patches, by adjusting the hyperparameter $K$ to cover more tokens.
Comparison: STRAP-ViT outperformed state-of-the-art defenses (e.g., JEDI, ODDR, PatchCleanser, Smoothed ViT) by a significant margin (e.g., 78.8% vs. 74.3% for ODDR on 40x40 patches).

5. Significance and Future Work

Efficiency: The computational cost of STRAP-ViT is minimal compared to the ViT backbone itself, as JSD calculation and transformations are $O(Nd)$, whereas transformer blocks are significantly more complex. It requires zero additional training cost.
Practicality: As a plug-and-play module, it can be deployed immediately in safety-critical systems (autonomous driving, surveillance) without retraining expensive foundation models.
Future Directions: The authors suggest extending this token-segregation approach to Large Vision Language Models (LVLMs) and other token-based architectures to secure the next generation of multimodal AI.

In conclusion, STRAP-ViT provides a highly effective, lightweight, and robust solution to the critical vulnerability of ViTs against physical adversarial patches, bridging the gap between theoretical security and practical deployment.