💻 computer science

Improving Adversarial Robustness of Zero-Shot CLIP with Confidence-Aware Weighting

This paper proposes Confidence-Aware Weighting (CAW), a novel method that enhances the adversarial robustness of zero-shot CLIP models by prioritizing uncertain samples through a confidence-aware loss and preserving semantic consistency via feature alignment, thereby outperforming state-of-the-art approaches in both robustness and memory efficiency.

Original authors: Nikoo Naghavian, Mostafa Tavassolipour

Published 2026-07-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nikoo Naghavian, Mostafa Tavassolipour

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers can "see" and "read" at the same time, learning from the entire internet to understand that a picture of a golden retriever matches the words "a happy dog." This is the magic of Vision-Language Models, like the famous CLIP. They are like super-smart students who have read every book and seen every photo online, allowing them to guess what's in a picture even if they've never seen that specific type of animal before. However, just like a human who might be tricked by a clever optical illusion, these AI models have a secret weakness. If you add a tiny, invisible amount of "noise" to an image—like sprinkling a few grains of digital sand that the human eye can't see—the computer can suddenly get completely confused, thinking a cat is a toaster. This is called an "adversarial attack," and it's a major problem because it means these powerful tools can be easily fooled in real-world situations.

The big question scientists are asking is: How do we make these models tougher without making them forget everything they already know? Usually, to teach a computer to be robust, you have to show it thousands of these tricky, noisy images during training. But there's a catch: if you treat every single image the same way, the computer wastes time studying the easy ones (which it already understands) and might still miss the really tricky ones that need the most help. It's like a teacher spending the whole class reviewing a math problem the student already solved perfectly, while ignoring the one problem that's actually causing them to fail the test.

This paper introduces a clever new strategy called Confidence-Aware Weighting (CAW) to fix this problem. The researchers realized that not all tricky images are created equal. Some images are so confusing that the model barely guesses correctly, while others are only slightly shaky. Instead of giving every confusing image the same amount of attention, CAW acts like a smart tutor who pays extra attention to the students who are struggling the most.

Here is how the method works, using a simple analogy: Imagine the AI model is a student taking a test.

The "Confidence-Aware" Part: When the student gets a question wrong or is unsure, the teacher (the CAW system) says, "Okay, this is a hard one! Let's focus on this." It gives extra weight to the images where the model is least confident. If the model is already pretty sure about an image, the teacher says, "You've got this, let's move on." This ensures the model spends its energy fixing its biggest weaknesses rather than re-learning what it already knows.
The "Feature Alignment" Part: The second part of the trick is making sure the student doesn't forget what they learned in previous classes. When the model looks at a tricky, noisy image, CAW checks to see if the "internal picture" it sees matches the "clean picture" it saw before. It's like telling the student, "Even though this photo is blurry, remember that it's still a dog, not a toaster." This keeps the model from getting confused and losing its original knowledge.

The researchers tested this idea on a massive collection of image datasets, including TinyImageNet and 14 others ranging from pets and flowers to medical scans. They pitted their new method against other top-tier techniques using some of the strongest "attacks" available, including a powerful automated test called AutoAttack. The results were promising: the CAW method didn't just survive the attacks; it actually performed better than the previous best methods. For example, on average across 15 different datasets, it improved the model's ability to stay correct under attack by about 2% compared to the runner-up, while also using less computer memory.

What's really cool is that the model didn't just get better at fighting off attacks; it also got better at looking at normal, clean pictures. The researchers found that by focusing on the "hard" examples, the model learned a more stable way of seeing the world. When they looked at how the model "paid attention" to different parts of an image, they saw that before the training, the model was easily distracted by the noise, looking at the background or random pixels. After using CAW, the model learned to focus on the actual object, even when the image was being attacked.

In short, this paper suggests that by being a little more selective about which confusing examples to study, we can build AI that is both smarter and tougher. It's a step toward making these powerful vision-language models reliable enough to be used in the real world, where things aren't always perfect and sometimes try to trick us. While the method is currently focused on the image part of the model and tested mostly on white-box attacks (where the attacker knows the model's secrets), it opens a new door for making AI more resilient without sacrificing its ability to learn new things.

Technical Summary: Improving Adversarial Robustness of Zero-Shot CLIP with Confidence-Aware Weighting

Problem Statement
Vision-language models (VLMs), particularly CLIP, demonstrate impressive zero-shot generalization across diverse tasks. However, they remain highly vulnerable to adversarial attacks, where imperceptible perturbations cause significant prediction shifts. While adversarial training is a standard approach for improving robustness, existing methods applied to large-scale VLMs face two primary limitations:

Inefficient Optimization: Current approaches treat all adversarial examples equally during training, regardless of their difficulty or stability. This ignores the fact that samples with low robust confidence (unstable predictions) contribute more to vulnerability than highly confident samples.
Resource and Performance Trade-offs: State-of-the-art methods like PMG-AFT and TGA-ZSR often require high memory usage or struggle to maintain robust accuracy under strong attacks (e.g., AutoAttack) without compromising clean accuracy or generalization.

Methodology: Confidence-Aware Weighting (CAW)
The authors propose Confidence-Aware Weighting (CAW), an adversarial fine-tuning framework designed to enhance the zero-shot robustness of pre-trained CLIP models while preserving their generalization capabilities. The method introduces a composite loss function ( $L_{total}$ ) comprising three terms:

Standard Contrastive Loss ( $L_{CE}$ ): The baseline cross-entropy loss used to align image and text embeddings.
Confidence-Aware Loss ( $L_{CA}$ ): This component addresses the inefficiency of treating all samples equally. It scales the Kullback-Leibler (KL) divergence between the prediction distribution of the fine-tuned model on adversarial inputs ( $P_{adv}$ $P_{a d v}$ ) and the frozen pre-trained model on clean inputs ( $P_{clean}$ $P_{c l e an}$ ).
- Mechanism: The KL divergence is weighted by $(1 - P_{adv, i, y_i})$ , where $P_{adv, i, y_i}$ is the predicted probability of the true class for the adversarial sample.
- Effect: This weighting assigns higher importance to "hard" adversarial examples where the model is uncertain (low confidence), forcing the model to focus on stabilizing these vulnerable cases. The authors specifically utilize $KL(P_{adv} \parallel P_{clean})$ rather than the reverse, arguing it better penalizes incorrect high-probability predictions in the adversarial distribution.
Feature Alignment Regularization ( $L_{Reg}$ ): To prevent overfitting and the forgetting of pre-trained knowledge, this term minimizes the $\ell_2$ $ℓ_{2}$ distance between the image features of the adversarial input processed by the fine-tuned encoder and the same input processed by the frozen pre-trained encoder.
- Distinction: Unlike methods like FARE that align clean frozen representations with adversarial fine-tuned ones, CAW aligns adversarial representations from both encoders to stabilize feature representations during the fine-tuning process.

The total objective is formulated as:
$L_{total} = L_{CE} + \alpha \cdot L_{CA} + \beta \cdot L_{Reg}$

Key Contributions

Novel Weighting Mechanism: The paper introduces a confidence-aware weighting strategy tailored for zero-shot VLMs, shifting focus to unstable, low-confidence adversarial samples to improve robustness without sacrificing clean accuracy.
Dual-Loss Architecture: The combination of a confidence-scaled KL divergence and a feature-level regularization term creates a balance between learning robust features and retaining semantic knowledge from pre-training.
Efficiency: The proposed method achieves state-of-the-art performance while utilizing less memory than competing baselines (PMG-AFT and TGA-ZSR).

Experimental Results
The authors evaluated CAW on the TinyImageNet dataset for fine-tuning and tested zero-shot performance on 15 additional datasets spanning general object recognition, fine-grained classification, scene recognition, domain-specific tasks, and medical imaging.

AutoAttack ( $\epsilon = 1/255$ ): CAW achieved the highest average robust accuracy (33.51%) across 15 datasets, outperforming PMG-AFT (31.55%) and TGA-ZSR (31.63%). It showed particular strength on datasets like CIFAR-10 and Caltech-101.
PGD-100 and CW Attacks: Under these attacks, CAW surpassed PMG-AFT in both robust and clean accuracy. For instance, under PGD-100, CAW achieved an average robust accuracy of 34.91% compared to PMG-AFT's 33.20%, while also improving clean accuracy (53.75% vs. 47.28%).
Memory Efficiency: CAW required approximately 15,986 MB of memory, which is lower than PMG-AFT (18,449 MB) and significantly lower than TGA-ZSR (21,227 MB).
Analysis:
- Confidence Analysis: Scatter plots revealed that after fine-tuning, the correlation between clean and robust confidence increased, with a shift toward the diagonal, indicating enhanced stability. The method effectively reduced "Clean Only" samples (correct on clean, wrong on adversarial) without creating over-confident misclassifications.
- Attention Maps: Visualizations showed that CAW helped the model maintain consistent, semantically meaningful attention on object-relevant regions for both clean and adversarial examples, whereas pre-fine-tuning models often attended to irrelevant background noise under attack.

Significance and Claims
The paper claims that CAW represents a significant step forward in securing zero-shot VLMs against adversarial threats. By explicitly targeting the model's uncertainty, the method learns more transferable and robust features. The authors emphasize that their approach improves robustness under strong attacks (AutoAttack, PGD, CW) while maintaining lower memory footprints, making it a practical solution for large-scale models.

Limitations
The authors acknowledge several limitations:

The method is currently specific to the CLIP architecture and has not been tested on other VLMs.
It only addresses perturbations in the image encoder, leaving the text encoder unaddressed despite its importance in VLMs.
Evaluation is limited to standard white-box attacks; black-box, adaptive attacks, and larger perturbation radii were not explored.

Technical Summary: Improving Adversarial Robustness of Zero-Shot CLIP with Confidence-Aware Weighting

More like this