🤖 AI

Beyond Attack Success Rate: A Multi-Metric Evaluation of Adversarial Transferability in Medical Imaging Models

This paper argues that relying solely on Attack Success Rate (ASR) is inadequate for evaluating adversarial robustness in medical imaging models, demonstrating through a systematic study of CNNs and Vision Transformers that a multi-metric framework incorporating perceptual quality and perturbation magnitude is essential for a comprehensive assessment of adversarial risk.

Original authors: Emily Curl, Kofi Ampomah, Md Erfan, Sayanton Dibbo

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Emily Curl, Kofi Ampomah, Md Erfan, Sayanton Dibbo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Why "Winning" Isn't Everything

Imagine you are a security guard at a high-tech museum (the Medical AI). Your job is to spot forgeries (adversarial attacks) on paintings.

For years, the only way to test if your security system was good was to ask one simple question: "Did the fake painting fool the guard?"

If the answer was Yes, the test was a "Success." If No, it was a "Failure." This is what researchers call Attack Success Rate (ASR). It's a simple "Pass/Fail" grade.

This paper argues that this "Pass/Fail" grade is dangerously incomplete.

The authors say: "Just because a forger fooled the guard doesn't tell us how they did it. Did they use a tiny, invisible scratch? Did they paint over the whole thing with a thick, obvious layer of mud? Did they use a cheap marker or a master artist's brush?"

In the world of medical AI, knowing only that an attack worked isn't enough. We need to know the cost of the attack and how obvious it is to a human doctor.

The Cast of Characters

To test this, the researchers set up a massive experiment involving:

The Paintings: Four different types of medical images (skin spots, eye scans, tissue samples, and chest X-rays).
The Guards (Models): Two different types of AI "guards."
- The Old School Guards (CNNs): These are like traditional detectives who look at clues one by one (like looking at the nose, then the eyes, then the mouth).
- The New School Guards (Vision Transformers/ViTs): These are like detectives who look at the whole picture at once, understanding how the nose relates to the eyes in a single glance.
The Forgers (Attacks): Seven different methods used to trick the AI, ranging from subtle tweaks to heavy-handed distortions.

The Experiment: The "Magic Trick" Test

The researchers tried to trick the AI guards using these forgers. They measured three things:

Did it work? (ASR): Did the AI get the diagnosis wrong?
How ugly is it? (Perceptual Metrics): If a human looked at the image, would they notice the change? (Measured by how much the image looks like the original).
How hard did they try? (Perturbation Strength): How much digital "noise" or change was added to the image?

The Shocking Discovery: The "Decoupling"

Here is the main finding, explained with an analogy:

Imagine you are trying to break into a house.

Scenario A: You pick the lock with a tiny, invisible tool. The door opens. (High Success, Low Effort, Invisible).
Scenario B: You kick the door down with a sledgehammer. The door opens. (High Success, High Effort, Very Obvious).

In the past, researchers thought that if you kicked the door down harder (increased the Perturbation Strength), you were more likely to succeed. They assumed there was a direct link between "how hard you hit" and "did the door open."

This paper found that link is broken.

They discovered that how well the attack works (ASR) has almost nothing to do with how ugly the image looks or how much "noise" was added.

You can have a tiny, invisible change that tricks the AI 100% of the time.
You can have a massive, ugly change that also tricks the AI 100% of the time.
Conversely, a huge, ugly change might fail completely, while a tiny one succeeds.

The Analogy: It's like trying to guess if a magician will pull a rabbit out of a hat. The researchers found that the size of the hat (perturbation strength) or how messy the magician looks (image quality) tells you nothing about whether the rabbit will actually appear (the attack success).

The New School vs. The Old School

The study also compared the two types of AI guards:

The Old School (CNNs): They were a bit unpredictable. Some were easy to trick, others were hard. If you tricked one, it didn't guarantee you could trick the next one.
The New School (ViTs): They were surprisingly uniform. If you could trick one of these new AI models, you could almost certainly trick any of them. It's like if you found a master key for one brand of lock, it opened every single lock of that brand. This is a huge risk because it means an attacker only needs to study one model to defeat them all.

Why This Matters for Doctors

In a hospital, a doctor looks at an X-ray.

If the AI is tricked by a tiny, invisible change, the doctor might not even notice the image has been tampered with. The AI gives a wrong diagnosis, and the patient gets the wrong treatment.
If the AI is tricked by a huge, obvious change, the doctor might say, "Hey, this image looks weird, I'll ignore the AI and look myself."

The paper warns us: We cannot rely on "Did the AI fail?" as our only alarm system.

Because the "Success Rate" is disconnected from "Image Quality," an attacker could theoretically create an attack that is invisible to the human eye but deadly to the AI. If we only look at the success rate, we might think we are safe because the attacks are "weak" (low noise), when in reality, they are perfectly effective.

The Bottom Line

The authors are saying: "Stop using a single ruler to measure a complex problem."

To keep medical AI safe, we need a Multi-Metric Framework. We need to measure:

Did it work?
How much did it change the image?
How much effort did it take to create?

Only by looking at all these factors together can we truly understand the risk and build systems that are safe for patients.

1. Problem Statement

Deep learning models, particularly Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), are increasingly deployed in medical image analysis. However, these systems are vulnerable to adversarial attacks—small, imperceptible perturbations that cause misclassification.

Current research relies heavily on Attack Success Rate (ASR) as the primary evaluation metric. ASR is a binary metric indicating only whether an attack succeeded or failed. The authors argue that ASR is insufficient because:

It ignores perturbation strength (how large the change is).
It ignores perceptual image quality (whether the change is visible to a human).
It fails to capture the cost/overhead of generating the attack.
It provides an incomplete picture of transferability (whether an attack crafted on one model works on another), especially when comparing heterogeneous architectures like CNNs and ViTs.

The core research gap is the lack of a cohesive framework analyzing the correlation between ASR and other metrics (PSNR, SSIM, L2 magnitude) across diverse medical datasets and architectures.

2. Methodology

The authors conducted a large-scale systematic empirical study involving the following components:

Datasets: Four medical image datasets were used:
- PathMNIST: Histopathology (9 classes).
- DermaMNIST: Dermatology (7 classes).
- RetinaMNIST: Ophthalmology/OCT (5 classes).
- CheXpert: Chest X-ray (14 classes, multi-label).
Models: Seven models representing two architectural families:
- CNNs: VGG-16, ResNet-50, DenseNet-121, Inception-v3.
- ViTs: DeiT, Swin Transformer, ViT-B/16.
- Note: All models were initialized with ImageNet pre-trained weights and used without fine-tuning to isolate architectural differences.
Adversarial Attacks: Seven attack methods were evaluated:
- Single-step: FGSM.
- Iterative: PGD, I-FGSM.
- Momentum-based: MI-FGSM, NI-FGSM, VMI-FGSM.
- Ensemble: Gradient aggregation across all surrogates.
Perturbation Budgets: Five $L_\infty$ bounded budgets ranging from near-imperceptible ( $\epsilon = 2/255$ ) to clearly visible ( $\epsilon = 32/255$ ).
Evaluation Metrics:
- ASR: Percentage of misclassified inputs.
- Perceptual/Quality: SSIM (Structural Similarity Index) and PSNR (Peak Signal-to-Noise Ratio).
- Perturbation Magnitude: L2 norm.
Experimental Scale: The study generated 3,500 experimental configurations (4 datasets $\times$ 5 budgets $\times$ 7 attacks $\times$ multiple surrogate/target pairs).
Analysis: Pearson and Spearman correlation coefficients were calculated to determine the relationships between ASR and the other metrics.

3. Key Contributions

Multi-Metric Framework: The paper proposes moving beyond ASR by integrating perceptual (SSIM, PSNR) and magnitude (L2) metrics into adversarial evaluation.
Cross-Architecture Analysis: It provides a comprehensive comparison of adversarial behavior between CNNs and the emerging ViT architectures in the medical domain.
Discovery of Decoupling: The study empirically demonstrates that ASR is statistically decoupled from perceptual quality and perturbation magnitude.
Transferability Insights: It reveals distinct transferability patterns between CNNs (asymmetric) and ViTs (uniformly high).

4. Key Results

A. ASR is Decoupled from Perceptual Metrics

The most significant finding is that ASR has near-zero correlation with SSIM, PSNR, and L2 magnitude.

Correlation Coefficients: Across all models, the correlation between ASR and perceptual metrics was negligible ( $|r| \le 0.17$ , $|\rho| \le 0.15$ ).
Implication: A high ASR does not necessarily mean a large perturbation or poor image quality. Conversely, a low ASR does not guarantee a "clean" image. Two attacks can have identical success rates but vastly different visual impacts.
Internal Consistency: In contrast, perceptual metrics (SSIM, PSNR, L2) are strongly correlated with each other ( $|r| \ge 0.87$ ), confirming they measure the same underlying "distortion" property.

B. Architecture Differences in Transferability

ViTs: Showed uniformly high transferability. Black-box attacks between different ViT models achieved ASR between 91.1% and 93.4%, regardless of the specific surrogate-target pair.
CNNs: Showed asymmetric transferability.
- Easiest Target: Inception-v3 (81.2–82.4% black-box ASR).
- Hardest Target: ResNet-50 (72.0–73.5% black-box ASR).
CheXpert Anomaly: CNN black-box ASR on CheXpert dropped significantly to 52.8%, likely due to the multi-label nature of the dataset complicating CNN-to-CNN transfer. However, ViTs achieved 100% ASR on CheXpert in both white-box and black-box settings.

C. Effect of Perturbation Budget

Increasing the perturbation budget ( $\epsilon$ ) from $2/255$ to $32/255$ (a 16-fold increase) resulted in marginal gains in ASR:

CNNs: Black-box ASR only rose from 75.5% to 79.0%.
ViTs: Black-box ASR remained flat between 91.4% and 92.8%.
This confirms that simply increasing perturbation strength does not linearly improve attack success, further validating the decoupling of ASR from perturbation magnitude.

D. Ensemble Attacks

The Ensemble attack method (aggregating gradients from multiple surrogates) was the most effective, narrowing the gap between white-box and black-box ASR to just 1.1%.

5. Significance and Implications

Clinical Risk Assessment: Relying solely on ASR is dangerous in medical AI. An attack could be highly successful (high ASR) while introducing subtle, invisible distortions (high SSIM/PSNR) that a radiologist might miss, or conversely, a visible distortion might fail to fool the model.
ViT Vulnerability: The uniform high transferability of ViTs suggests that if an adversary compromises one ViT model, they can likely compromise any other ViT model in the ecosystem with high success, posing a systemic risk.
Future Evaluation Standards: The authors argue for a multi-metric evaluation framework that considers:
1. Efficacy: (ASR)
2. Methodology: (Perturbation type)
3. Overheads: (Perceptual quality, L2 cost)
Conclusion: ASR alone is an inadequate indicator of adversarial robustness. A holistic risk assessment must account for the "cost" of the attack (visual degradation) alongside its success rate to ensure patient safety in clinical deployments.