🤖 machine learning

Exploring Adversarial Watermarking in Transformer-Based Models: Transferability and Robustness Against Defense Mechanism for Medical Images

This paper investigates the vulnerability of Vision Transformers (ViTs) in dermatological image analysis to adversarial watermarking, demonstrating that while these models are highly susceptible to accuracy drops from such attacks, their robustness can be significantly improved through adversarial training.

Original authors: Rifat Sadik, Tanvir Rahman, Arpan Bhattacharjee, Bikash Chandra Halder, Ismail Hossain, Mridul Banik, Jia Uddin

Published 2026-02-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Rifat Sadik, Tanvir Rahman, Arpan Bhattacharjee, Bikash Chandra Halder, Ismail Hossain, Mridul Banik, Jia Uddin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "Master Chef" and the "Invisible Salt"

An Explanation of Adversarial Watermarking in Medical AI

Imagine you are a world-class Master Chef. You have a legendary ability to taste a soup and instantly know exactly which ingredients are in it—even if they are subtle. This is how modern Vision Transformers (ViTs) work in medical imaging. They are like these Master Chefs; they look at a medical image (like a photo of a skin lesion) and "taste" the global patterns to decide if it’s a harmless mole or something dangerous like melanoma.

But there is a problem: a sneaky "saboteur" has figured out a way to ruin your sense of taste without you even noticing.

1. The Saboteur’s Trick (Adversarial Watermarking)

Imagine someone sprinkles a tiny, microscopic amount of a special, invisible salt into your soup. To a regular person, the soup looks and smells exactly the same. But to your expert palate, that one tiny grain of "invisible salt" completely confuses you. Suddenly, you taste a delicious tomato soup and insist it’s actually a bowl of chocolate pudding.

In this paper, the researchers used a method called PGD (Projected Gradient Descent). Think of PGD as a high-tech "invisible salt generator." It adds tiny, pixel-level changes to medical images. To a doctor looking at the photo, the skin looks perfectly normal. But to the AI "Chef," these tiny changes act like a confusing watermark that tricks the model into making a massive mistake—like calling a dangerous cancer "benign."

2. The "Global" Weakness (Why Transformers are Vulnerable)

The researchers found something surprising. While older AI models (called CNNs) are like chefs who focus on one ingredient at a time (looking at small, local spots), the new Vision Transformers (ViTs) are chefs who try to taste everything at once to get the "big picture."

Because the Transformer is trying to understand how every single part of the image relates to every other part, it is actually more sensitive to the sabotage. If the saboteur messes with the relationship between the pixels, the Transformer gets completely lost. The study showed that while the Transformer was the smartest chef on a "clean" dish (94.4% accuracy), it was the easiest to fool, crashing down to a terrible 27.6% accuracy when the "invisible salt" was added.

3. The Solution: "Training the Palate" (Adversarial Training)

So, how do we stop the saboteur? We don't just give the chef better ingredients; we give them tougher training.

The researchers used a technique called Adversarial Training. Imagine instead of only practicing with perfect, clean soups, the Master Chef spends months practicing with "sabotaged" soups. They learn to recognize the taste of that invisible salt and learn to ignore it.

By training the AI on these "tricky" images, the Transformer became incredibly tough. Its accuracy on the sabotaged images jumped from a failing 27.6% all the way up to a stellar 90.0%.

The Bottom Line

This research is a "wake-up call" for the medical world. It tells us that while the newest, most powerful AI models (Transformers) are brilliant at diagnosing diseases, they have a "blind spot" that hackers could exploit. However, by teaching these models to expect and resist these "invisible" attacks during their training, we can make them safe enough to trust in real hospitals.

Technical Summary: Exploring Adversarial Watermarking in Transformer-Based Models

1. Problem Statement

As deep learning transitions from Convolutional Neural Networks (CNNs) to Vision Transformers (ViTs) in medical imaging (specifically dermatology), new security vulnerabilities emerge. While ViTs excel at capturing global contextual relationships—crucial for identifying subtle diagnostic features in skin lesions—their reliance on global self-attention mechanisms makes them uniquely susceptible to adversarial perturbations. The core problem addressed is the vulnerability of ViTs to adversarial watermarking (imperceptible perturbations designed to fool models) and the potential for these attacks to misclassify critical medical conditions, such as mistaking malignant melanoma for benign lesions.

2. Methodology

The researchers employed a comparative experimental framework to test the robustness of ViTs against traditional CNN architectures.

Attack Mechanism: The study utilized Projected Gradient Descent (PGD) to generate adversarial watermarks. PGD is an iterative optimization method that maximizes the model's prediction loss by adding small, constrained perturbations ( $\epsilon$ ) to the input image, ensuring the changes remain imperceptible to the human eye.
Models Evaluated:
- Transformer-based: Vision Transformer (ViT).
- CNN-based: ResNet-50 and VGG16.
Dataset: A dermatological dataset comprising five skin disease classes: Atopic dermatitis, Eczema, Herpes, Nevus, and Melanoma.
Defense Mechanism: The authors implemented Adversarial Training. This involves augmenting the training dataset with adversarial examples generated via PGD, forcing the model to learn robust features that are invariant to such perturbations.
Workflow: The process involved data collection $\rightarrow$ preprocessing (normalization/augmentation) $\rightarrow$ baseline training $\rightarrow$ adversarial attack generation $\rightarrow$ adversarial training $\rightarrow$ comparative evaluation.

3. Key Contributions

Vulnerability Analysis of ViTs: The paper identifies that the global attention mechanism in ViTs, while beneficial for accuracy, creates a larger attack surface for distributed, subtle perturbations compared to the localized feature extraction of CNNs.
Transferability Investigation: The study explores how adversarial watermarks impact different architectures, highlighting the risks in heterogeneous medical AI environments.
Robustness Benchmarking: It provides a quantitative comparison of how different architectures (ViT vs. CNN) respond to both clean and adversarial inputs.
Defense Validation: It demonstrates the efficacy of adversarial training specifically for transformer-based medical imaging models.

4. Results and Discussion

The empirical results revealed a significant disparity between model performance on clean versus adversarial data:

Baseline Performance (Clean Images): ViT and ResNet-50 performed excellently, achieving 94.4% and 94.2% accuracy, respectively, significantly outperforming VGG16 (84.0%).
Adversarial Vulnerability: Under PGD attack, ViT’s accuracy plummeted to 27.6%, making it far more fragile than ResNet-50 (70.0%) and VGG16 (54.4%). This confirms that ViTs are highly sensitive to perturbations that disrupt inter-patch relationships.
Effectiveness of Defense: Adversarial training proved highly successful for the transformer model. ViT’s accuracy on adversarial images recovered from 27.6% to 90.0% (a +62.4% improvement). ResNet-50 and VGG16 also saw improvements, but the relative gain for ViT was most substantial.

5. Significance

This research is highly significant for the deployment of AI in clinical settings. It highlights a critical "security-performance trade-off": the very mechanism that makes ViTs superior for medical diagnosis (global attention) also makes them a high-risk target for adversarial manipulation. By proving that adversarial training can effectively bridge this security gap, the study provides a roadmap for developing "safe" transformer-based diagnostic tools that can withstand malicious attempts to alter medical outcomes.